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Executive  Summary 


This  is  the  final  report  for  Data  Capture  During  Strong  Angel  (DARPA  contract  #  DAAB07-99-C-C201), 
describing  MITRE’ s  evaluation  of  the  CommandNet  collaborative  groupware  tool  used  during 
RIMPAC2000,  a  Third  Fleet  multinational  exercise  that  included  Strong  Angel,  a  humanitarian  aid  and 
disaster  relief  (HA/DR)  scenario  involving  both  military  and  non-government  organizations  (NGO). 
During  this  effort,  MITRE  worked  in  partnership  with  researchers  at  the  Center  for  the  Management  of 
Information  at  the  University  of  Arizona  who  developed  CommandNet,  and  with  the  US  Navy  Third  Fleet. 


CommandNet  is  a  web-based  electronic  logbook  used  to  capture  data  and  observations  that  were  shared 
within  and  across  command  centers  afloat  on  the  US S  Coronado  and  ashore  at  a  simulated  refugee  camp  on 
the  island  of  Hawaii.  Users  during  Strong  Angel  included  military  personnel  and  members  of  a  United 
Nations  team  involved  with  management  of  the  refugee  camp.  The  exercise  provided  an  opportunity  to 
collect  data  across  groups  of  users  collaborating  over  time  in  an  operational  environment,  analyze  the 
technological,  social  and  organizational  processes  characterizing  such  collaborative  interactions,  and 
evaluate  effectiveness  of  a  collaborative  tool  in  an  operational  civil-military  scenario 


CommandNet  was  instrumented  for  automatic  capture  of  usage  patterns,  and  in  addition,  a  participant 
observer  collected  qualitative  data  on-board  the  Coronado  and  at  the  refugee  site.  The  analysis  of  both 
quantitative  and  qualitative  data  on  CommandNet  usage  collected  during  a  month  of  RIMPAC,  including 
the  five  days  of  Strong  Angel,  is  presented  in  this  final  report. 


Quantitative  data  came  from  both  automatic  data  capture  and  from  user  questionnaires.  Automatic  data 
capture  included  actual  content  of  the  CommandNet  logs,  such  as  text  entries  with  associated  timestamps 
and  user  identities,  plus  instrumented  capture  of  events  from  the  CommandNet  server,  such  as  navigation 
from  window  to  window,  or  use  of  a  feature.  Survey  questionnaires  were  used  to  get  background  about 
users,  such  as  rank,  specialization,  and  experience  using  computers  and  collaborative  tools,  and  to  elicit 
data  about  CommandNet  features  and  overall  user  reaction  to  the  tool. 


Qualitative  information  about  usage  was  gathered  through  comments  in  the  questionnaires,  interviews  with 
users,  and  observations  by  a  trained  ethnographer  who  was  on-board  and  ashore  throughout  the  exercise 
and  was  eventually  accepted  as  a  fellow  participant.  The  open-ended  interviews  were  conducted  to  gather 
contextual  information  about  the  situations  in  which  CommandNet  was  being  used,  and  how  CommandNet 
fit  into  a  larger  picture  of  what  users  were  trying  to  accomplish  during  the  exercise..  The  emphasis  of  the 
observations  was  on  how  people  interacted  with  the  collaborative  technology  and  with  each  other,  and  on 
organizational  processes  surrounding  use  of  CommandNet.  Daily  field  notes  were  maintained,  and 
recorded  interviews  were  transcribed. 


The  two  types  of  data  illuminated  each  other  in  several  ways.  Observations  about  usage  were  supported  by 
hard  data.  Quantitative  usage  patterns  out  of  the  norm  could  be  explained  by  field  notes  indicating  that 
some  event  had  disrupted  regular  operations.  Event  tracking  could  be  used  to  verify  user  comments. 
Together  the  two  types  of  data  provide  a  picture  of  usage,  utility  and  user  reactions,  all  situated  within  the 
social  and  organizational  dynamics  of  collaborative  activity. 


Virtually  all  users  found  CommandNet  useful  and  generally  better  than  hardcopy  logbooks  previously  used. 
They  appreciated  the  availability  of  a  common  repository  of  current  information  that  was  easy  to  access, 
even  remotely,  and  easy  to  monitor  and  review.  The  tool  was  especially  useful  during  the  overlap  at  shift 
changes,  when  the  outgoing  staff  could  update  the  incoming  staff  by  reviewing  the  shift’s  logged  entries. 
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Because  of  its  web  interface,  CommandNet  was  easy  to  use,  which  was  good  because  training  was  limited 
or  non-existent.  Many  users  just  learned  by  example,  watching  another  user  make  an  entry,  or  modeling 
their  entry  on  an  existing  one.  One  factor  that  negatively  affected  usability  was  the  interaction  between 
CommandNet  and  the  communications  network  infrastructure  that  sometimes  caused  delays  in  response 
time  when  an  entry  was  submitted.  The  screen  might  go  blank  for  minutes,  causing  users  to  repeatedly 
click  on  Submit  and  to  inadvertently  generate  duplicate  entries.  Another  usability  problem  that  frustrated 
users  was  a  log  refresh  that  could  occur  while  the  user  was  scrolling  through  entries  and  would  reset  the  log 
screen  to  the  top  of  the  entries. 


Very  few  users  exploited  any  of  the  advanced  features  supported  by  CommandNet,  including  marking  an 
entry  for  Importance  (a  priority  level)  or  with  a  Category  (a  classification  label),  grouping  related  entries 
under  Headlines  for  analysis.  Searching  by  keyword  or  by  category,  or  setting  User  Preferences  such  as 
refresh  rate.  In  general,  users  converged  on  the  basic  log  entry  submission  and  review  process  required  for 
their  work,  and  did  not  explore  further  use. 


Collaboration  is  an  inherently  social  activity  that  does  not  lend  itself  to  controlled  experiments,  especially 
in  operational  environments  involving  hundreds  of  people,  millions  of  dollars  of  equipment,  and  a  mission 
to  protect  refugees.  Also,  CommandNet  was  just  one  of  many  technologies  used  during  the  exercise  to 
complement  face-to-face  interactions,  and  it  is  not  easy  to  isolate  its  contribution  from  those  of  other  tools. 


However,  some  general  observations  can  be  drawn.  As  a  tool  for  asynchronous  coordination, 

CommandNet  provided  shared  information  for  situational  awareness,  support  for  smooth  transitions  over 
shift  changes,  and  a  historical  record  for  replay  and  accountability.  It  was  also  useful  for  synchronous 
coordination  among  distributed  participants.  Logs  were  used  for  communication  between  ship  and  shore 
and  between  the  refugee  base  camp  and  remote  command  center.  They  were  useful  helping  a  group  resolve 
ambiguities,  clarify  interpretations  and  reach  a  unified  view,  a  process  which  was  critical  to  the  civil- 
military  coordination  required  for  the  exercise. 


For  robust  evaluation,  it  is  necessary  to  have  complementary  data,  both  quantitative  and  qualitative,  and  to 
use  complementary  levels  of  analysis,  such  as  usage  patterns,  human-computer  usability  factors,  and 
organizational  structures  and  processes.  It  is  also  essential  to  have  complementary  organizational 
perspectives,  such  as  CMI’s  view  as  CommandNet  developers,  MITRE’ s  view  as  collaboration 
technologists  and  evaluators,  and  the  Third  Fleet’s  view  as  operational  users. 


Transition  of  a  collaborative  tool  into  an  operational  environment  is  easier  if  the  capabilities  of  the  tool 
match  the  requirement  of  the  mission.  The  basic  element  of  CommandNet  is  the  log  which  is  a  particularly 
good  fit  to  Navy  operations  because  maintaining  logs  is  an  integral  process  that  everyone  knows  how  to  do. 
Successful  transition  also  requires  organizational  commitment  to  collaborate,  to  use  the  tool  and  to  evaluate 
its  use.  Such  commitment  by  the  Third  Fleet  made  this  study  possible. 


For  the  same  tool,  usage  patterns  differ  by  user  roles  and  missions.  In  RIMP AC/Strong  Angel,  eight 
different  logs  yielded  eight  different  usage  patterns.  It  is  also  apparent  that  usage  patterns  evolve  over  time, 
and  eventual  familiarity  and  increased  expertise  might  prompt  more  users  to  try  the  advanced  features  of 
CommandNet  they  had  originally  ignored. 
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Introduction 


This  report  describes  an  evaluation  by  MITRE  of  utility  and  usability  of  the  CommandNet  collaborative 
groupware  tool  in  the  context  of  RIMPAC.  RIMPAC  is  a  major  Third  Fleet  multinational  exercise  held 
every  two  years  with  participating  forces  from  the  United  States,  Australia,  Canada,  Chile,  Japan,  Korea 
and  the  United  Kingdom.  Held  in  June  of  2000,  RIMPAC2000  was  made  up  of  several  exercise  phases, 
including  Strong  Angel,  a  humanitarian  aid  and  disaster  relief  (HA/DR)  scenario  involving  both  military 
and  non-government  organizations  (NGO). 


CommandNet  (CN),  developed  by  researchers  at  the  Center  for  the  Management  of  Information  at  the 
University  of  Arizona,  is  a  web-based  distributed  information- sharing  environment  that  supported  various 
command  centers  both  on-board  the  USS  Coronado  and  on-shore  at  a  simulated  refugee  camp  on  the  island 
of  Hawaii.  CN  is  an  electronic  logbook  used  to  capture  data  and  observations  that  were  shared  within  and 
across  command  cells  to  contribute  to  situational  awareness.  Via  CommandNet,  users  can  enter  information 
into  logs  (viewable  by  others  in  their  logbook  community),  combine  log  entries  into  groups  (‘Headlines’), 
and  access  relevant  entries  using  key  word  and  category  searches.  Users  during  Strong  Angel  included 
military  personnel  and  civilian  disaster  relief  staff,  such  as  members  of  a  United  Nations  team  involved 
with  management  of  the  refugee  camp. 


Because  of  MITRE’ s  leadership  of  the  Evaluation  Working  Group  (EWG)  of  DARPA’s  Collaboration, 
Visualization  and  Information  Management  (CVIM)  programs,  and  its  experience  in  evaluating 
collaboration  technologies,  DARPA  tasked  MITRE  to  document  and  analyze  the  usage  of  CommandNet  in 
RIMPAC2000,  focusing  primarily  on  Strong  Angel.  An  objective  of  the  study  was  to  provide  useful 
feedback  that  might  guide  CommandNet  usage  and  training  decisions  by  the  Third  Fleet  and  future  design 
choices  by  developers. 


Period  of  performance  of  this  effort  was  January  through  September  2000.  Prior  to  the  summer  exercise, 
MITRE  staff  became  familiar  with  the  evolving  CN  environment,  contributed  iteratively  to  the  design  and 
development  of  the  CN  version  deployed  in  RIMPAC,  and  specified  instrumentation  capabilities  needed 
for  automatic  capture  and  analysis  of  usage  patterns.  During  the  exercise  itself,  a  MITRE  observer  (JB) 
collected  qualitative  data  on-board  the  USS  Coronado  and  at  the  refugee  site.  The  analysis  of  both 
quantitative  and  qualitative  data  on  CN  usage  collected  during  the  exercise  is  presented  in  this  final  report. 


Methodology 

We  collected  both  quantitative  and  qualitative  data.  Below,  we  discuss  details  of  the  data  and  how  the  data 
were  gathered.  We  also  describe  how  the  data  were  processed  and  how  the  various  types  data 
complemented  each  other  to  provide  a  more  accurate  picture  for  evaluation. 


Data  Collection 

The  initial  phase  of  this  effort  was  focused  on  becoming  familiar  with  CommandNet  general  capabilities 
and  providing  insight  in  usability  features  to  CMI.  For  several  months  preceding  RIMPAC2000,  MITRE 
used  evolving  versions  of  CommandNet  and  provided  feedback  to  CMI  developers  on  usability  of  the  tool 
and  on  instrumentation  necessary  to  gather  usage  data  needed  for  assessment.  These  interactions  included 
regular  pilot  testing  sessions  in  which  MITRE  and  CMI  exploited  CommandNet  design  features  and 
stressed  the  environment.  MITRE  developed  its  own  scripts  and  tools  for  eventual  processing  of  data 
generated  by  CommandNet  during  the  summer  exercise.  MITRE  also  developed  a  questionnaire  to  elicit 
background  information  and  feedback  from  Strong  Angel  and  RIMPAC  users  of  CommandNet. 
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During  RIMPAC2000,  and  Strong  Angel  in  particular,  MITRE  collected  two  types  of  data:  quantitative  and 
qualitative. 


Quantitative  data  came  from  both  automatic  data  capture  and  from  user  questionnaires.  Automatic  data 
capture  included  (a)  content  of  the  CommandNet  logs  themselves,  and  (b)  instrumented  capture  of  events 
from  the  CommandNet  server.  The  log  content  constituted  of  an  ongoing  record  of  what  was  entered, 
associated  with  time  of  entry  and  the  user  identity  or  role  of  the  author.  Instrumented  event  capture  on  the 
server  recorded  events  such  as  navigation  from  one  screen  to  another  and  usage  of  features  such  as  Search 
and  Headlines.  All  collected  data  were  maintained  at  the  appropriate  security  classifications. 


Not  available  in  any  automated  way  were  client  side  events  such  as  scrolling  or  opening/closing  of  browser 
windows.  That  is,  it  was  not  possible  to  have  an  accurate  fine-grained  view  of  what  a  user  might  be 
looking  at  all  points  in  time. 


Survey  questionnaires  were  used  to  get  background  about  users  (e.g.,  rank,  specialization,  computer 
experience,  experience  with  collaborative  tools,  etc.),  and  to  elicit  data  about  CommandNet  features  and 
overall  user  reaction  to  the  use  of  the  tool.  Users  were  asked  to  rate  each  of  the  CommandNet  features  on  a 
5-point  Likert  scale,  with  additional  space  for  comments.  Questions  on  the  overall  user  experience  were  a 
combination  of  Likert  scale  questions  and  open-ended  questions  designed  to  elicit  feedback  on  usability, 
user  satisfaction  and  ideas  for  tool  improvements.  See  Appendix  A  for  the  actual  survey  used. 


Qualitative  information  about  usage  was  gathered  through  interviews  and  observations  by  a  trained 
ethnographer  (JB).  Open-ended  interviews  were  conducted  to  gather  contextual  information  about  the 
situations  in  which  CommandNet  was  being  used.  Because  of  time  constraints,  format  of  the  interviews 
varied  considerably,  from  short  impromptu  conversations  to  scheduled  tape-recorded  sessions.  Interviews 
focused  on  how  CommandNet  fit  into  a  larger  picture  of  what  users  were  doing  during  the  exercise.  The 
interviews  also  included  questions  on  how  participants  decided  to  use  CommandNet  as  well  as  how  they 
were  introduced  to  the  tool. 


The  trained  observer  came  to  the  exercise  several  days  before  the  beginning  of  Strong  Angel  and  remained 
throughout  the  rest  of  RIMPAC2000.  In  addition  to  observing  CommandNet  interaction  onboard  the  USS 
Coronado,  JB  also  participated  in  the  on-shore  component  of  Strong  Angel.  She  came  to  be  considered  a 
regular  member  of  the  Strong  Angel  community,  and  participated  in  meetings,  operations  and  community 
activities.  The  emphasis  of  the  observations  was  on  how  people  interacted  with  the  collaborative 
technology  and  with  each  other,  and  on  organizational  processes  surrounding  the  use  of  CommandNet. 


Data  Processing  and  Analysis 

Both  quantitative  and  qualitative  data  were  prepared  to  facilitate  analysis.  Quantitative  data  from  the  HCI 
surveys  were  tabulated  in  an  Excel  spreadsheet.  The  automatically  captured  events  from  the  server  logs 
were  parsed  into  spreadsheets,  as  well,  for  data  manipulation,  table  creation,  and  graphical  displays. 


Qualitative  data  from  surveys  were  organized  by  topic.  Interviews  and  notes  were  transcribed  and 
organized  by  content.  These  data  were  then  distributed  to  the  research  team  to  provide  context  for 
analyzing  the  quantitative  data. 
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Interpretation  of  observations  and  interviews  conducted  during  the  exercise  provided  deeper  insight  into  the 
meaning  of  patterns  found  in  the  server  logs.  Conversely,  the  quantitative  data  supported  theories  and 
observations  of  particular  behavior.  In  addition,  data  from  the  server  logs  were  used  to  verify  that  user 
feedback  reflected  actual  use  of  the  tool.  Thus,  comparing  data  collected  through  various  methods 
informed  the  team  and  laid  a  solid  foundation  for  analysis  and  evaluation. 


The  research  team  wanted  to  understand  the  usage  of  CommandNet  during  Strong  Angel/RIMPAC  2000 
and  whether  the  tool  features  supported  this  usage  appropriately.  Contents  of  the  logs  were  annotated  for  a 
high-level  view  of  usage  and  style.  Participant-tool  interactions  were  profiled  over  time  to  look  for  changes 
in  behavior  and  adoption  or  rejection  of  particular  features.  Ease  of  use  of  CommandNet  was  investigated 
particularly  since  training  had  been  minimal  and  informal. 


Results 

First,  to  provide  appropriate  context,  we  discuss  the  CommandNet  user  population  in  both  Strong  Angel 
and  RIMPAC  exercises.  Then  we  provide  an  overview  of  the  utility  and  usability  of  CommandNet, 
describing  basic  features  as  well  as  some  more  advanced  features.  For  more  detailed  results,  see  Appendix 
B.  A  full  CommandNet  Usability  Report  is  included  as  Appendix  C. 


User  Population  Characteristics 

The  Strong  Angel  user  population  for  CommandNet  consisted  of  both  military  and  non  government 
organizations:  US  Marine  Corps  personnel,  UN  team  members,  reservists,  and  ComThirdFleet 
representatives.  The  RIMPAC  user  population  was  all  military:  primarily  US  Navy  with  some  Army  and 
Marine  representatives,  and  military  personnel  from  non-US  countries. 


All  of  the  surveyed  members  of  the  user  population  had  some  experience  in  using  computers,  and  all  had 
used  some  type  of  collaborative  tool  previously  (e.g.,  email,  shared  directories;  some  had  even  used 
GroupSystems^).  In  the  past,  participants  had  used  a  variety  of  other  media  for  reporting  observations  and 
accessing  information  including  email,  paper-based  logbooks,  and  older  (non-web-based)  versions  of 
CommandNet. 


Participation  differed  relative  to  user  roles  within  the  operations  efforts.  Those  directly  responsible  for 
routine  communications  were  often  the  ones  to  make  entries  while  those  higher  up  in  the  hierarchy 
(leaders)  had  authority  to  make  decisions  about  what  constituted  appropriate  usage  of  the  tool.  With  a  tool 
relatively  new  to  many,  ‘appropriate  usage’  was  something  not  always  agreed  upon  or  understood,  and  so 
adoption/diffusion  across  the  Strong  Angel  population  was  relatively  uneven. 


utility  and  Usability 

Overall,  CommandNet  users  found  the  tool  better  than  a  hardcopy  logbook.  There  was  general  agreement 
that  CommandNet  was  effective  as  a  common  data  repository,  making  information  easily  accessible  within 
the  HA/DR  context. 


'‘Before,  we  would  get  the  information  and  put  it  on  a  slip  of  paper  and  thumbtack  it 
to  a  piece  of  wood  or  something.  Not  everyone  would  have  access  to  this  piece  of 
information.  We  never  had  a  problem  of  collecting  the  information  before  -  just  a 
question  of  how  we  disseminated  it.  ” 

-  CommandNet  user 


1  GroupSystems,  a  group  support  system  designed  to  facilitate  group  interaction,  was  developed  by  CMI. 
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We  surveyed  24  out  of  1 10  CommandNet  users.  The  low  response  rate  was  due  to  time  constraints  and 
tight  schedules.  Over  half  the  participants  surveyed  indicated  that  they  found  CommandNet  to  be  a  useful 
tool.  See  figure  1,  below.  The  participants  were  also  enthusiastic  about  the  ability  to  scroll  back  through 
the  logs  and  review  or  monitor  data  submitted  by  others.  This  ability  to  review  past  entries  was  particularly 
useful  during  shift  changes  when  the  overlapping  shifts  could  use  CommandNet  as  a  focal  point  in 
discussing  events  that  had  occurred. 


Basic  Usage 

The  simplest  and  most  commonly-used  feature  of  CommandNet  was  entry  creation/submission.  With  little 
or  no  training,  participants  found  CommandNet  easy  to  learn  and  use  for  logging  entries.  Figure  2,  below, 
shows  that  70%  of  the  surveyed  users  found  CommandNet  easy  to  use.  No  one  found  it  difficult  to  use. 


CommandNet  was  also  easy  to  administer.  Administrators  were  given  access  to  creating  user  accounts, 
editing  entries,  and  adding  new  categories  to  the  logs. 


One  of  the  biggest  complaints  about  the  use  of  CommandNet  was  the  slowness  of  the  system.  The 
performance  issues  were  confounded  by  network  latency.  When  a  user  submitted  an  entry,  the  screen 
would  go  blank,  sometimes  up  to  several  minutes,  before  the  updated  list  of  entries  would  reappear. 


'dt  freezes  up  ...  When  you  need  it  the  most,  you  are  unable  to  use  it. 


Log  refresh  was  also  distracting  and  frustrating  to  users  reviewing  the  history  of  events;  when  the  screen 
refreshed,  the  entry  list  would  be  reset  to  the  latest  entry.  Although  CommandNet  provided  multiple  ways 
of  controlling  the  refresh  (the  user  could  change  the  refresh  rate  or  disable  it  altogether),  only  about  one 
fourth  of  the  user  population  ever  took  advantage  of  these  features.  Since  few  participants  received  any 
training  in  CommandNet,  the  refresh  features  were  probably  overlooked. 


When  an  entry  was  created,  the  user  had  the  option  of  adding  an  associated  category  and  an  importance 
level.  A  category  provided  a  visible  label  while  importance  allowed  the  users  to  prioritize  an  entry. 
Depending  on  user  preferences,  both  category  and  importance  could  be  displayed  alongside  the  logged 
entries. 


Categories  potentially  made  it  easier  for  users  to  identify  meaningful  entries  quickly;  the  advanced  search 
feature  (see  below)  also  enabled  users  to  search  the  entry  list  by  category.  Categories  could  be  created  by 
anyone  with  the  right  administrative  permissions.  What  we  found  was  that  categories  were  used  in  just 
three  of  eight  logs.  Those  responsible  for  deciding  how  a  log  would  be  used  either  dictated  that  categories 
should  be  used  -  and  then  they  were  used  -  or  did  not  make  a  decision  regarding  category  usage  -  and  so 
they  were  not  used.  A  new,  untrained  user  was  more  likely  to  learn  by  example  and  create  entries  similar 
to  those  already  in  the  log;  if  categories  were  not  used,  the  new  user  would  not  use  them  either.  In  some 
cases,  the  existing  categories  did  not  necessarily  correspond  to  categories  that  participants  found  useful, 
and  an  entry  would  be  submitted  without  a  category.  Users  commented  that  the  category  feature  was  not 
used  as  well  as  it  should  have  been.  If  categories  had  been  agreed  upon  in  advance  and  their  use  discussed 
by  the  participant  teams,  then  they  may  have  shown  a  higher  utility  for  log  review  and  monitoring. 


Unlike  categories  which  could  be  created  by  an  administrator,  importance  levels  were  predefined.  A  user 
could  choose  from  ‘Routine’  (the  default),  ‘Priority’,  ‘Immediate’,  or  ‘Flash’.  However,  the  importance 
levels  were  displayed  as  ‘R’,  ‘P’,  ‘O’,  and  ‘Z’.  Some  users  said  they  never  used  importance  because  they 
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did  not  understand  their  definitions.  Others  simply  did  not  associate  the  single-letter  display  with  a  level  of 
importance.  A  few  users  commented  that  there  was  no  need  to  prioritize  entries  since  they  were  involved 
in  nothing  more  than  an  exercise.  Importance  levels  were  used  in  just  three  of  eight  logs.  It  is  interesting 
to  note  that  these  three  logs  were  not  the  same  subset  of  logs  in  which  categories  were  used. 


Advanced  Features 

While  CommandNet’s  basic  design  was  “Drop  Dead  Simple”,  it  also  contained  a  number  of  advanced 
features  that  were  not  used  during  the  exercises.  These  advanced  features  were  generally  unobtrusive.  The 
search,  headline,  and  user  preferences  features  were  recessed;  to  access  them,  the  user  would  have  to  click 
on  a  menu  item  which  led  to  a  different  web  page. 


The  results  of  the  survey  indicate  that  the  search  and  headline  feature  were  not  used  by  participants.  The 
event  records  show  that  about  one  fourth  of  all  users  viewed  those  pages,  a  few  users  even  attempted  to 
perform  a  test  search  or  create  a  headline,  but  no  searches  were  saved  and  no  headlines  submitted. 


Users  were  more  likely  to  change  their  preferences,  including  what  information  was  displayed,  the  time 
offset,  and  the  refresh  rate. 


The  fact  that  advanced  features  were  used  so  infrequently  may  be  explained  by  several  reasons,  most  being 
related  to  the  short  duration  of  the  exercises  under  ‘emergency’  conditions,  without  much  lead  planning, 
and  the  frustration  experienced  by  users  due  to  the  system’s  slowness.  Had  the  tool  been  in  use  for  a  longer 
time  during  non-exercise  periods,  there  might  have  been  more  exploration  /  experimentation  with  different 
features,  which  may  or  may  not  have  been  adopted  by  the  general  population. 


User  Communities:  Scenarios  and  Usage  Patterns 

Each  CommandNet  log  was  created  for  a  particular  community  of  users.  Decisions  regarding  the  usage  of 
each  log  were  mandated  by  different  people.  In  some  cases,  no  decisions  were  made,  and  participants 
tended  to  follow  the  example  of  those  who  used  the  tool  before  them.  As  a  result  of  these  variations,  each 
log  was  used  quite  differently. 


In  this  section,  we  provide  some  context  by  discussing  the  characteristics  of  each  user  community.  Then 
we  briefly  summarize  how  each  log  was  used.  We  distinguish  the  user  communities  participating  in  the 
Strong  Angel  exercises  from  those  participating  in  other  RIMPAC  exercises  because  the  participant  groups 
differed  greatly  and  their  use  of  the  tool  varied  as  well.  For  more  details  on  tool  usage,  see  Appendix  B: 
Detailed  Results. 


Strong  Angel  User  Community 

Although  there  were  three  distinct  CommandNet  logs  created  and  used  during  Strong  Angel,  there  was  too 
little  data  collected  over  the  short  exercise  to  warrant  separate  discussions  of  each  log.  Below,  we  profile 
the  Strong  Angel  logs  as  one. 


CMOC  User  Community 

Context 


8  of  21 


The  MITRE  Corporation 


Collaborative  Data  Capture  during  Strong  Angel  and  RIMPAC  2000 


The  term  ‘CMOC’  normally  refers  to  the  ‘Civil-Military  Operations  Center’,  a  specially  designed 
conference  center  aboard  the  Coronado.  ‘CMOC’  was  also  used  during  Strong  Angel  exercise  to  refer  to 
the  group  of  representatives  from  “civilian”  (UN)  and  military  (Navy,  Marines  and  Army)  organizations 
that  used  the  CMOC  space  to  discuss  and  plan  their  cooperative  efforts. 


During  Strong  Angel  on  the  Big  Island,  a  large  tent  was  set  up  to  support  CMOC  group  activities.  This 
‘CMOC  Ashore’  contained  tables,  chairs,  and  laptops.  It  housed  the  frequent  meetings  of  the  ‘CMOC’ 
group,  and  was  also  utilized  as  a  daytime  shelter  from  the  dust  by  some  participants.  The  laptops  in  the  tent 
were  connected  to  a  LAN  and  ran  collaborative  and  internet  software,  including  CommandNet.  At  various 
times  throughout  the  day,  someone  sitting  at  the  table  would  use  the  laptop  in  front  of  him  or  her  to  review 
and/or  make  entries  to  the  CMOC  log(s)  in  CommandNet.  Occasionally  military  officers  would  come  into 
the  tent  specifically  to  use  CommandNet.  Other  people  accessing  the  logs  with  any  regularity  included  the 
Civil  Affairs  officers  who  were  also  stationed  inside  the  CMOC  tent,  and  a  couple  of  UN  communications 
people  in  the  UN  tent  who  had  CN  on  their  machines  hooked  up  to  the  same  LAN. 


At  the  start  of  Strong  Angel’s  ashore  component,  one  CommandNet  log  was  established  for  the  ‘CMOC’ 
and  another  for  the  Marines.  However  the  Marine  officers  were  too  busy  with  other  responsibilities  to 
explore  using  the  software.  The  CMOC  log  attracted  a  range  of  people,  and  eventually  two  other  logs  were 
spun  off  -  a  Civil  Affairs  log,  and  a  ‘CMOC  Afloat’  log,  each  in  an  effort  to  separate  low  level  tactical 
information  from  items  of  interest  to  higher-ups.  The  ‘Civil  Affairs’  log  was  created  to  support 
communication  between  Civil  Affairs  officers  at  the  base  and  refugee  camps;  its  use  was  limited  because  it 
depended  on  a  radio  antenna  for  its  internet  connection  that  was  frequently  not  working.  The  ‘CMOC 
Afloat’  log  was  set  up  when  the  HF  radios  intended  for  ship  to  shore  communication  failed,  and 
CommandNet  became  the  only  ship-to- shore  communications  link  for  the  CMOC  group. 


Usage  Profile  of  Strong  Angel  Logs 

Strong  Angel  participants  differed  from  participants  in  RIMPAC  2000  exercises  in  that  Strong  Angel  users 
received  more  training  from  CMI  staff.  Overall,  they  were  pleased  with  the  training  they  received  and 
found  the  tool  easy  to  use. 


was  good.  They  showed  me  the  basics,  and  if  I  had  any  questions,  I  knew  who 
was  around  to  ask.  I  mean  if  I  was  in  my  office  by  myself,  1  could  probably  figure  it 
out  on  my  own.  I  think  it's  very  simple  so  you  don't  need  a  lot  of  training  which  is 
what  I  like  about  it.  ” 


Strong  Angel  participants  used  CommandNet  mostly  to  record  situation  reports  and  tracking  events,  both 
people  and  resources.  There  was  also  significant  use  for  troubleshooting,  chat,  request  for  action, 
acknowledgment,  and  testing.  The  category  feature  was  used  consistently  in  the  Civil  Affairs  log,  about 
half  the  time  in  the  CMOC  log,  and  not  at  all  in  the  CMOC  Afloat  log.  Surveyed  participants  had  mixed 
reactions  to  the  use  of  categories. 


'‘Could  be  useful  if  we  used  it.  Problem  is  we  don't  use  it  well. 


Although  the  search  feature  was  not  used,  some  of  the  participants  thought  it  was  a  good  idea  to  have. 


"Reusability  of  saved  searches  is  an  important  feature. 
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Overall,  Strong  Angel  participants  surveyed  said  that  CommandNet  was  easier  to  use  than  other 
comparable  systems.  They  found  it  useful,  appropriate  for  the  exercise,  and  easy  to  use.  Most  commonly 
requested  features  included  chat  or  a  log  reserved  for  side  conversations,  and  the  ability  to  add  attachments. 


RIMPAC  User  Communities 

There  were  five  logs  used  during  the  RIMPAC  exercises:  one  for  each  of  the  BWC,  JCCC,  Intel,  CFACC, 
and  CECG  communities. 


BWC  in  the  JOC 
Context 

The  user  community  for  the  Battle  Watch  log  is  primarily  the  officers  standing  the  Battle  Watch  in  the  Joint 
Operations  Center  (JOC).  The  purpose  of  the  watch  is  to  support  the  situational  awareness  of  the  Battle 
Watch  Captain  (BWC),  who  is  the  "eyes  and  ears”  of  the  Admiral  and  the  Chief  of  Staff  (COS).  In 
addition  to  the  BWC  and  the  Assistant  BattleWatch  Captain  (ABWC),  six  or  seven  other  officers  are  also 
stationed  in  the  JOC  to  support  the  BWC  in  this  capacity.  These  “anchors”  have  various  backgrounds 
including:  Underwater  Sea  Warfare  (USW  =  submarines),  Intel,  Ground  Forces  (Army),  and  Coalition 
(other  countries). 


Information  arrives  in  the  JOC  via  a  variety  of  sources:  Microsoft  NetMeeting  Chat,  high  frequency  radio, 
telephone,  CNN  TV,  various  collaborative  applications  used  by  anchors,  maps,  briefings,  web  sites, 
software  displaying  common  operating  picture  maps,  etc.  Different  people  within  the  room  attend  to 
different  sources  so  that  some  messages  are  received  by  all,  many  by  several  people,  and  others  by  only  a 
single  person.  As  relevant  items  of  information  are  identified  by  participants  on  the  watch,  they  are 
mentioned  and  discussed,  either  with  one  or  two  other  people  close  by,  or  spoken  more  loudly  across  the 
room.  Much  of  the  “information”  that  comes  in  is  incomplete  and/or  ambiguous  and  needs  further 
clarification  before  it  is  considered  to  be  useful  or  actionable.  Under  the  discretion  of  the  BWC,  such 
information  is  entered  into  the  (CommandNet)  BattleWatch  log  by  the  ABWC.  The  text  of  the  message  is 
usually  dictated  by  the  BWC  to  the  ABWC.  The  CommandNet  log  is  projected  onto  a  portion  of  one  of 
three  large  screens  in  the  front  of  the  semi-darkened  room;  these  many  of  the  other  software  applications 
that  are  in  use  concurrently. 


The  Battle  Watch  is  maintained  around  the  clock  by  officers  on  six-hour  shifts.  Before  an  official  shift 
change,  officers  from  the  incoming  shift  will  arrive  about  a  half  an  hour  early  to  talk  with  their  outgoing 
counterparts  about  what  has  been  happening  to  gain  current  situation  awareness.  At  this  time, 
CommandNet  is  often  maximized  on  the  screen  and  used  as  a  review  tool  by  scrolling  back  through  the  last 
six  to  twelve  hours  of  logged  entries.  Officers  at  other  locations  around  the  ship  also  have  access  to  the  log 
—  most  who  do  only  monitor  the  log  and  do  not  make  entries. 


Usage  Profile  of  the  BWC  Log 

The  Battle  Watch  Captain’s  log  was  used  as  an  operational  log.  During  Strong  Angel,  entries  were  mainly 
about  trouble  shooting,  personnel,  tasking,  and  scheduling.  During  the  RIMPAC  war  exercises,  entries 
shifted  towards  sitreps,  tracking  of  forces,  reports,  and  casualties.  A  portion  of  these  entries  contained 
information  from  other  logs  (CFACC,  Intel,  and  CECG). 
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The  BWC  log  was  used  throughout  the  entire  month,  during  both  Strong  Angel  and  RIMPAC  exercises. 
There  were  typically  about  22  entries  a  day,  but  activity  increased  from  about  20  entries  a  day  to  60  during 
the  exercises.  Entries  before  the  war  exercises  were  made  by  the  Battle  Watch  Captain.  During  the 
exercises,  the  Battle  Watch  Captain  dictated  the  entries  to  the  Assistant  Battle  Watch  Captain. 


Before  using  CommandNet  as  a  means  of  reporting  observations  and  accessing  information,  participants 
had  used  a  local  log  (the  greenbook),  email,  and  a  previous  version  of  CommandNet  referred  to  as  “the 
collaborative  logbook.”  There  was  infrequent  usage  of  the  category  and  importance  feature.  More 
advanced  features  (search  and  headlines)  were  not  used  at  all.  Most  everyone  learned  the  basic  tool  usage 
from  watching  someone  else,  and  they  all  thought  CommandNet  was  relatively  easier  to  use  than  past 
methods  of  recording  observations  and  accessing  information. 


Overall,  BWC  users  surveyed  judged  CommandNet  to  be  somewhat  useful,  somewhat  easy  to  use,  and  very 
appropriate  to  their  job.  What  users  liked  most  about  CommandNet  were  ease  of  use  and  its  value  as  a 
permanent  record  of  events.  They  commented  on  how  easy  it  was  to  add  entries  and  how  easy  and  useful 
the  scroll  back  feature  was. 


JCCC  User  Community 

Context 

The  JCCC  (Joint  C4I  Control  Center,  or  tech  support)  log  was  unrelated  to  the  RIMPAC  or  Strong  Angel 
exercises.  The  JCCC  log  was  primarily  used  in  a  small  room  on  the  ship  containing  several  people  and 
numerous  computers  and  monitors.  The  people  using  it  (making  entries)  are  enlisted  people  who  are 
providing  tech  support  and  system  maintenance  to  people  on  the  ship.  MITRE’ s  participant  observer  was 
not  able  to  spend  a  lot  of  time  in  this  room,  and  consequently  we  do  not  know  a  lot  about  what  went  on  in 
the  room. 


The  use  of  the  log  by  this  group  was  established  by  an  operations  deputy.  Previously,  they  had  been  using 
a  Microsoft  Word  document  for  this  purpose.  CommandNet  was  preferred  to  Word  because  it  made  it 
impossible  for  the  entries  to  be  changed  after  they  had  been  submitted.  One  of  the  staff  served  as 
administrator  for  the  JCCC  log. 


Usage  Profile  of  the  JCCC  Log 

The  JCCC  log  was  created  for  use  in  the  Joint  Command  Control  Center.  The  JCCC  log  was  used  for 
technical  support:  logging  troubleshooting  events,  maintenance,  and  system  and  network  status.  The  users 
in  the  JCCC  community  were  all  regulars  and  were  accustomed  to  logging  entries  in  other  tools  like  an 
Excel  spreadsheet.  Word  document,  or  local  logs  (greenbook).  None  of  the  users  interviewed  had  received 
any  training;  they  learned  to  login  and  make  entries  from  someone  on  the  previous  watch.  The  log  was  in 
use  for  16  days.  The  activity  level  in  the  log  was  fairly  constant  throughout  its  use.  A  typical  user 
averaged  about  seven  entries  a  day. 


Compared  to  methods  utilized  in  the  past,  JCCC  users  found  CommandNet  ''easy  to  use''  and,  according  to 
one  person,  "about  the  best  I’ve  seen."  They  liked  the  fact  that  the  entries  were  numbered  because  they 
could  refer  to  them  by  the  number.  Indeed,  about  3%  of  the  entries  contained  references  to  other  entries. 
Interviewed  users  also  liked  the  automatic  time  stamps  associated  with  each  entry.  Another  well-liked 
feature  was  the  ability  to  scroll  back  through  the  history  of  logged  events.  Users  reportedly  reviewed  the 
log  at  the  beginning  of  their  shift  and  sometimes  also  at  the  end. 
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Not  one  user  complained,  during  our  interviews,  about  the  slowness  of  the  system.  That  was  surprising 
considering  that  5%  of  the  entries  were  duplicates  resulting  from  hitting  the  submit  button  more  than  once 
before  the  system  had  time  to  update  the  logs.  The  JCCC  log  had  a  higher  percentage  of  duplicate  entries 
than  any  other  log.  More  than  one  user  requested  the  ability  to  edit  entries;  there  were  numerous 
typographical  errors. 


Other  than  for  simple  data  entry,  the  JCCC  user  community  did  not  use  CommandNet  for  anything  else. 
The  easy-to-use  category  and  importance  features  were  not  used  to  annotate  entries,  and  none  of  the  more 
advanced  features  (search,  headlines)  were  used  at  all.  A  few  users  viewed  and  changed  the  user 
preference  settings,  but  the  refresh  feature  was  not  used. 


The  JCCC  community  found  CommandNet  somewhat  easier  than  other  tools.  Overall,  it  was  rated  as 
somewhat  useful,  somewhat  appropriate,  and  somewhat  easy  to  use. 


Intel  User  Community 

Context 

mitre’s  participant  observer  did  not  have  access  to  the  spaces  in  which  most  of  the  Intel  officers  worked. 
Upon  questioning,  she  ascertained  that  there  were  often  between  eight  and  nine  people  in  such  a  room,  and 
discussion  of  what  incoming  messages  meant  was  frequently  a  prerequisite  before  any  entry  was  made  in 
the  log. 


Usage  Profile  of  the  Intel  Log 

The  Intel  log  contained  more  entries  than  any  other  log.  It  was  used  heavily  during  the  RIMPAC  exercises. 
In  fact,  activity  increased  drastically  from  fewer  than  10  entries  a  day  to  a  range  between  65  and  125  a  day. 
In  addition  to  making  entries  in  the  Intel  log,  the  typical  Intel  user  monitored  the  BWC  log.  Before  the 
RIMPAC  war  exercises,  entries  consisted  of  reports,  general  information,  and  information  on  the  enemy 
forces.  During  the  war  exercises,  entries  consisted  of  sitreps  and  tracking  information.  Some  of  the  entries 
were  Intel  messages  pasted  directly  into  CommandNet. 


At  least  one  third  of  the  surveyed  population  was  reservist.  All  were  computer  users  and  familiar  with 
email  and  shared  directories.  Other  comparable  methods  for  making  observations  and  accessing 
information  were  Microsoft  Excel  spreadsheet,  Microsoft  Word,  local  logs  (greenbook),  and  older  versions 
of  CommandNet.  No  one  surveyed  had  received  any  training. 


Unlike  other  logs,  the  category  feature  was  used  heavily,  and  most  (67%)  found  it  both  useful  and  easy  to 
use.  The  importance  feature  was  used  regularly  in  the  Intel  logs,  too,  but  was  rated  as  less  than  useful. 


‘7  don't  prioritize  reading  based  on  that  since  I  read  them  as  they  come  in.  I  don't 
do  this  often  enough  to  remember  what  'O’  and  'P’  mean.  ” 


A  couple  of  users  made  several  searches  more  than  once.  Searches  were  rated  useful,  helpful,  and  easy  but 
were  considered  slow  and  not  so  intuitive.  One  person  suggested  that  it  might  be  easier  to  use  the  ‘Eind’ 
feature  of  the  browser.  The  headline  feature,  an  analytical  feature  for  grouping  entries,  was  not  used,  but  a 
few  people  explored  the  page. 
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Most  users  said  CommandNet  was  somewhat  easier  than  other  systems  and  not  any  faster.  Everyone 
interviewed  thought  it  provided  better  analysis.  Overall,  they  unanimously  found  it  useful,  appropriate,  and 
easy. 


Below  are  some  comments  about  what  the  Intel  community  liked. 


''auto  log  entry  time,  precedence  level,  originator,  as  much  space  as  needed  to  write 
whatever” 

"The  fact  that  it's  quick,  easy  to  use,  research  tool  for  current  events.  It's  a  log. 
Recap  events  go  back  and  figure  out  what's  going  on.  Keeps  key  nodes  on  ship 
apprised  of  what's  going  on.  ” 

"Very  intuitive  ui.  It  gives  me  access  to  real  time  info.  ” 


Requested  features  included  the  ability  to  force  refresh,  chat,  and  the  capability  of  adding  attachments. 

"I  would  like  more  info.  It  would  be  nice  to  have  a  way  to  ask  questions.  If  I  am 
unclear,  I  send  out  email.  The  log  is  not  meant  to  be  interactive.  ” 

CFACC  User  Community 

Context 

The  Coalition  Air  Force  Command  and  Control  (CFACC)  log  is  similar  in  many  respects  to  the  Battle 
Watch  log,  with  some  significant  differences.  It  is  primarily  used  in  one  section  of  a  large  open  room  —  the 
CAOC  (Coalition  Air  Operations  Center).  Other  sections  of  the  room  are  used  for  various  Intel  and 
planning  activities  related  to  air  operations.  Similarly,  there  are  multiple  screens  in  front,  but  they  are 
computer  monitors,  stacked  two  levels  high.  There  are  more  maps  of  different  kinds  and  fewer  other 
collaborative  applications. 


Most  of  the  people  in  the  CAOC  are  Navy  pilots.  The  Watch  Officers  are  junior  officers  (who  have  never 
done  a  watch  before);  they  sit  closest  to  the  computer  monitors.  The  action  in  the  CAOC  is  much  faster 
than  that  observed  in  the  JOC.  People  come  together  quickly,  have  short  conversations,  and  then  break  up 
their  conversation  formations  more  rapidly;  more  people  are  standing  (and  walking  in  and  out),  and  fewer 
are  sitting. 


The  explanation  given  for  why  the  CAOC  is  kept  separate  from  the  JOC,  and  why  there  is  no  Air 
Operations  Anchor  in  the  JOC,  is  that  the  CAOC  is  mostly  tactical,  while  the  JOC  is  (trying  to  be)  more 
operational.  In  general,  things  happen  a  lot  faster  in  the  CAOC  -  planes  fly  much  faster  than  ships  sail,  and 
everything  else  is  affected  by  that  difference. 


Usage  Profile  of  the  CFACC  Log 

The  Coalition  Forces  Air  Command  Control  Log  was  used  for  just  six  days  during  the  RIMPAC  exercises. 
There  was  little  activity  in  the  log,  about  44  entries  a  day.  Two  thirds  of  surveyed  participants  were 
regulars,  one  third  were  reservist.  All  were  users  of  email,  chat,  shared  directories,  and  document 
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management  software.  Tools  used  in  the  past  to  record  observations  and  access  information  included 
Microsoft  Word  and  local  logs,  or  the  greenbook.  Two  thirds  of  the  surveyed  population  received  no 
training  on  the  use  of  CommandNet. 


Categories  were  not  used,  and  none  was  available.  The  importance  feature  was  used  in  this  log  although 
some  users  chose  not  to  prioritize  their  entries:  the  exercise  was  ''not  really  a  warC  Advanced  features 
were  not  used  at  all.  And,  unlike  users  of  the  other  logs,  the  CFACC  group  did  not  record  their  shift 
changes. 


In  general,  CFACC  log  users  liked  the  automatic  timestamp  associated  with  each  entry.  They  liked  being 
able  to  see  other  people’s  entries  and  other  logs  (BWC). 


CECG  User  Community 

Context 

The  Exercise  Control  staff  used  a  CommandNet  log  during  the  RIMPAC  exercise.  They  sat  in  the  JOC 
(usually  two  people,  together)  and  used  their  log  in  only  that  single  location.  They  found  it  useful  for 
changing  watch  shifts  and  as  a  historical  record,  but  did  not  find  much  use  for  its  distributed  nature. 


Usage  Profile  of  the  CECG  Log 

The  CECG  log  was  created  for  the  Coalition  Exercise  Control  Group.  The  purpose  of  the  log  was  for 
recording  significant  exercise  events  and  operational  activities.  Everyone  surveyed  was  a  regular.  All  were 
experienced  computer  users  and  had  used  email,  chat,  and  shared  directories.  Most  had  used  Microsoft 
Word  in  the  past  for  similar  event  logging.  According  to  our  survey,  no  one  interviewed  received  any 
training  on  the  use  of  CommandNet.  The  log  was  in  use  for  eight  main  days.  All  entries  were  made  by  the 
CECG  watch  officer.  About  24  entries  were  made  a  day,  possibly  by  at  least  two  different  people  or  shifts 
per  day  (there  were  two  shift  changes  logged  per  day). 


CommandNet  was  used  by  the  CECG  user  community  mainly  to  log  entries.  People  interviewed  said  they 
used  the  scroll  back  feature  at  the  beginning  of  their  watch  to  review  previous  entries.  There  was  a  belief 
expressed  by  two  users  that  no  one  was  looking  at  the  logs,  although  there  was  some  evidence  that  there 
were  people  who  logged  in  just  to  monitor  the  logs.  No  features  were  used  other  than  change  in  user 
preferences.  The  lack  of  feature  use  could  be  attributed  to  the  lack  of  training  or  to  the  urgency  of  the 
situation. 


"When  it  gets  busy,  1  only  want  to  make  entries. 


Most  popular  positive  comments  about  CommandNet  were  regarding  numbered  entries  and  automatic 
timestamps.  Slowness  of  the  system  was  a  common  complaint.  According  to  one  participant,  the  slowness 
made  CommandNet  unreliable  as  a  means  of  communication:  "when  you  need  it  most,  you’re  unable  to  use 
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Discussion 

Social  and  Organizational  Implications 

Collaboration:  Tool  Use  is  Dispersed,  Not  Modular 

Human  collaboration  is  not  easily  separated  from  a  larger  social  context.  Using  a  collaborative  tool  is  only 
one  small  aspect  of  a  much  larger  ongoing  process.  People  collaborate  not  just  with  the  tool  but  in  a 
multiplicity  of  forms  and  in  an  ongoing  way.  Evaluating  the  utility  of  a  collaborative  tool  involves  looking 
beyond  human-computer  interactions,  in  order  to  identify  how  use  of  the  tool  changes  the  larger  context  of 
collaboration.  This  kind  of  evaluation  cannot  be  done  in  a  controlled  experiment  as  many  human-computer 
interaction  studies  are  done,  since  real  collaboration  is  always  socially  situated. 


Evaluating  a  collaborative  tool  to  gain  knowledge  of  its  impact  in  an  operational  setting  involves  looking  at 
a  much  larger  phenomenon.  The  current  case,  evaluating  CommandNet  in  an  operational  context,  situated 
the  use  of  the  tool  in  exercise  operations  involving  hundreds  of  people  and  millions  of  dollars  worth  of 
equipment.  In  this  context,  the  use  of  CommandNet  was  often  commensurate  with  phone,  radio  and 
physical  transportation.  People  used  CommandNet  at  least  once  a  day  and  often  several  times  or  more  an 
hour,  interspersing  tool  use  with  lots  of  other  collaborative  activity,  including  human-human  interactions 
and  utilizing  other  collaborative  software. 


To  bring  some  order  to  the  multifarious  ways  that  tool  use  affected  the  broad  social  context,  we  consider 
several  broad  categories  of  social  implications  of  CommandNet  use.  Eirst  we  consider  how  use  of  the  tool 
supported  coordination  over  time,  via  asynchronous  communication;  then  we  examine  how  use  of  the  tool 
supported  coordination  over  space,  or  synchronous  communication.  Einally,  we  consider  some  of  the 
issues  involved  with  adoption  and  diffusion  of  the  tool  in  the  available  population. 


Coordination  Over  Time:  Asynchronous  Communication 

Situational  Awareness 

Perhaps  the  simplest  use  of  CommandNet  is  as  a  reminder  of  what  has  occurred.  This  is  the  way  it  is  used 
to  support  situational  awareness  (SA),  its  primary  requirement  When  a  new  event  occurs,  people  can  refer 
back  to  the  tool’s  event  history  for  context  in  interpreting  the  current  situation.  What  was  significant  about 
this  was  that  CommandNet  was  designed,  presented,  and  used  as  a  log.  In  the  Navy,  the  ‘log’  already 
exists  as  a  communication  genre;  people  know  how  to  use  logs  and  so  were  readily  able  to  make  use  of  the 
tool  thanks  to  its  simple  design. 


Reference  for  Shift  Changes 

CommandNet  was  frequently  used  during  shift  changes.  In  an  operational  setting  (24/7),  people  take  turns 
filling  different  positions;  most  military  operations  involve  shifts,  and  the  shift  change  is  an 
organizationally  significant  event.  Shift  changes  are  when  organizational  continuity  is  enacted,  spanning 
individual  experience.  Those  who  are  not  present  during  certain  events  can  be  briefed. 


BattleWatch  shift  changes  are  a  prime  example.  When  the  incoming  BattleWatch  Captain  (BWC)  arrived, 
the  outgoing  BWC  would  open  CommandNet  and  review  the  log(s)  to  identify  significant  events  for 
discussion.  The  incoming  BWC  would  regard  the  log  as  an  indicator  of  recent  events  relevant  to  the  watch 
and  that  knowledge  would  form  the  context  for  events  to  come.  More  detailed  examination  of  how  shift 
changes  are  carried  out  is  important  to  future  efforts. 
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Log  as  Historical  Record 

Logs  are  maintained  as  a  historical  record,  useful  if  there  is  a  problem,  to  trace  back  for  increased  insight 
and  understanding  about  what  happened.  They  provide  support  for  social  accountability  which  is  important 
to  maintaining  organizational  order.  The  Navy  has  historically  maintained  logs  for  various  operations,  and 
it  is  because  of  the  Navy’s  familiarity  with  logs  that  CommandNet  was  created  to  be  used  as  a  log.  The 
importance  of  this  fact  should  not  be  overlooked;  collaborative  technologies  are  especially  effective  when 
they  are  introduced  into  a  population  already  used  to  communicating  in  certain  ways  that  the  technology 
can  then  support. 


Training  through  Learning  by  Example 

Especially  during  an  exercise,  many  of  the  participants  are  unfamiliar  with  their  positions.  People  who  are 
new  to  an  operation  often  learn  by  observing  and  imitating  what  they  see  around  them.  This  applies,  in 
particular,  to  entries  they  see  in  the  log.  Log  content  is  considered  by  many  novices  to  be  an  indication  of 
the  correct  or  proper  format  for  entries;  any  new  entries  they  make  are  likely  to  be  similar  in  form  and 
content  to  the  ones  that  are  already  there.  Since  training  is  acknowledged  as  a  key  aspect  of  exercises, 
although  it  is  often  undersupported  with  resources  and  planning,  the  importance  of  having  sample  entries 
and/or  templates  prepared  ahead  of  time  should  be  highlighted.  For  example,  there  probably  would  have 
been  more  usage  of  categories  if  the  initial  entries  had  employed  them. 


Coordination  Over  Space:  Synchronous  Communication  via  Log 

Distributed  Situational  Awareness 

As  a  web-based  tool,  CommandNet  was  used  to  support  coordination  distributed  over  space.  People  used 
the  log  to  stay  coordinated  with  others  who  were  physically  remote,  often  in  place  of  telephone,  radio,  or 
physical  transportation.  For  example,  Intel  analysts  located  on  three  different  decks  of  the  ship  could  work 
collaboratively  without  needing  to  walk  around  to  all  the  spaces.  CommandNet  was  perceived  as  an 
improvement  over  radio  communication  as  it  was  more  intelligible;  similarly,  watchstanders  at  CMOC  and 
refugee  camps  could  coordinate  without  walking  2  miles  in  dust  (or  by  a  half  hour  HMV  trip). 


Distributed  Sense-making 

In  an  operations  center,  messages  come  in  over  a  multiplicity  of  channels  and  are  often  incomplete  or 
ambiguous;  interpreting  the  messages  takes  a  collective  effort.  In  this  situation,  a  single  shared  view  is  one 
key  to  coordination.  In  the  Joint  Operations  Center,  the  log  is  displayed  in  the  front  of  the  room,  where 
people  (literally)  "share  a  common  view.”  This  provides  a  common  ground  for  interpreting  incoming 
messages.  Once  something  is  in  the  log,  it  is  agreed  to  be  "what  really  happened".  Prior  to  that  there  is 
confusion  and  sense-making  in  the  JOC  room.  The  same  sense-making  process  also  goes  on  in  the  JAOC 
and  the  Intel  spaces.  (''Oh  yes,  there’s  eight  to  nine  people  in  that  space,  and  they  all  discuss  what  should 
be  entered  for  each  entry  in  the  logC) 


The  sensemaking  process  is  further  accentuated  during  exercises  when  many  of  the  participants  are  not  yet 
trained,  and  are  trying  to  figure  out  what  is  happening.  What  winds  up  in  the  log  may  still  be  ‘wrong’  and 
need  to  be  fixed  later,  but  it  serves  as  a  common  indicator  of  agreed-upon  reality  that  everyone  can  then 
base  their  future  work  on,  and  by  doing  so,  stay  coordinated  with  each  other.  Through  this  process, 
personnel  construct  organizational  continuity/cohesion  —  not  only  with  respect  to  the  entries  in  the  log,  but 
with  respect  to  their  relationships  to  each  other  that  are  hammered  out  by  way  of  discussions  about  what 
should  go  in  the  log.  (E.g.,  this  may  be  why  humor  is  such  a  regular  feature  of  conversations  in  the  JOC.) 
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Increased  Social  Accountability 

Another  factor  in  social  and  organizational  processes  affected  by  the  use  of  CommandNet,  and  other 
distributed  collaboration  tools,  has  to  do  with  people’s  work  being  visible  at  a  distance  -  not  only 
geographically,  but  status-wise.  People  higher  up  in  the  hierarchy  can  more  easily  see  and  monitor  what 
people  at  lower  levels  are  doing.  From  one  perspective,  this  has  both  advantages  and  disadvantages  - 
leaders  may  want  more  detail,  but  also  may  have  to  struggle  not  to  get  mired  in  irrelevant  details.  For 
example,  during  RIMPAC,  the  Chief  of  Staff  said  he  was  not  interested  in  looking  at  the  logs  because  the 
level  of  detail  was  too  distracting  for  the  strategic  level  of  focus  he  needed  to  maintain. 


From  the  other  perspective,  it  also  has  positive  and  negative  effects.  On  the  one  hand,  it  is  a  form  of 
visibility  which  can  lead  to  greater  recognition  of  work  performed.  On  the  other  hand,  it  can  also  lead  to 
extra  work  around  making  a  good  impression,  as  demonstrated  by  the  numerous  requests  we  received  for 
enhanced  spell-checking  and  editing  capabilities.  Disadvantages  associated  with  distant  collaboration  are 
accentuated  by  power  differentials.  Sociologist  Judith  Perrolle  points  out  the  importance  of  informal 
“backstage”  regions  in  which  people  can  act  without  fear  of  being  seen  from  outside/above,  as  critical  to 
moral  development  as  discussed  by  Habermas.^  (E.g.,  informal  support  over  Microsoft  NetMeeting  during 
presentations  to  the  Admiral).  This  is  a  very  thorny  problem  with  important  arguments  on  both  sides,  and 
will  not  be  settled  easily  or  unilaterally  anytime  soon.  It  bears  significantly  on  the  difference  between  a 
formal  log  and  an  informal  chat,  and  is  a  good  reason  to  keep  them  intentionally  separate. 


Adoption  and  Diffusion 


There  were  conflicting  influences  on  the  adoption  and  diffusion  of  CommandNet  throughout  the  Strong 
Angel  and  RIMPAC  populations.  This  section  explores  how  these  influences  affected  tool  adoption  and 
diffusion  and  led  to  variations  in  usage  across  different  sub-populations.  In  addition  to  ease-of-use, 
influences  identified  include: 

•  (non-)mobility  of  participant’ s  role  in  the  exercise 

•  time  pressures  and  urgency  of  exercise  contingencies  on  leadership 

•  access  to  CMI  developers  and  support  personnel 


The  matter  of  relative  mobility  is  significant.  Some  people  (especially  those  responsible  for  routine 
communications)  had  jobs  which  kept  them  in  a  single  location  throughout  the  day.  Others  were  involved 
in  the  operations  of  setting  up  a  camp  and  roaming  two  sites  in  order  to  address  a  wide  range  of  issues; 
these  people  did  not  have  ongoing  access  to  a  networked  computer  that  would  readily  afford  them 
opportunities  to  access  the  collaborative  log(s).  Those  people  who  were  unfamiliar  with  the  tool  and  had 
the  time  to  learn  it  were  people  with  stationary  jobs  on  the  front  lines  of  information  reporting.  For  those 
with  mobile  responsibilities,  the  potential  usefulness  of  wireless  communications  was  fairly  clear;  changes 
are  anticipated  in  that  direction. 


During  exercises,  leadership  needed  to  be  much  more  concerned  with  fighting  a  war  or  securing  food, 
water  and  shelter  for  large  numbers  of  people  than  focused  on  the  complexities  of  adapting  to  a  new 
collaborative  tool.  To  put  the  tool  to  optimal  use  involves  a  certain  degree  of  advance  planning  -  i.e. 
agreeing  on  what  kinds  of  tracking  to  use  the  tool  for  and  in  what  formats,  etc.  Such  preparations  had  not 
always  been  made  in  advance,  and  so  the  usage  of  the  tool  was  sometimes  limited  by  this  fact. 


^  Perrolle,  J.  A.,  Privacy  and  Surveillance  in  Computer  Supported  Cooperative  Work  , 
http://www.ccs.neu.edu/home/perrolle/privacy.html 
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Another  factor  influencing  adoption  and  diffusion  was  access  to  the  developers  and  related  support 
personnel.  While  a  few  computer- savvy  individuals  took  it  upon  themselves  to  explore  and  experiment 
with  CommandNet,  others  needed  more  support  to  feel  comfortable  using  the  tool.  The  most  frequent  users 
of  CommandNet  during  Strong  Angel  turned  out  to  be  those  people  who  had  legitimate  reasons  to  be  in  the 
same  location  as  the  CMI  support  personnel  on  an  ongoing  basis.  As  a  result  of  this  close  proximity,  they 
could  easily  ask  impromptu  questions  and  get  immediate  assistance. 


Lessons  Learned 

Technical  Transition 

CommandNet  came  directly  from  a  technology  development  environment  and  was  installed  in  a  Sea-Based 
Battle  Laboratory  for  use  in  an  operational  exercise.  There  are  many  factors  that  affect  the  successful 
transition  of  technology  into  operational  use,  and  one  of  the  most  important  is  a  clear  and  early 
commitment  by  the  operational  organization  to  use  the  technology.  In  the  case  of  Strong  Angel,  the 
decision  to  include  the  web-based  CommandNet  tool  was  not  finalized  until  the  Coronado  was  underway 
from  San  Diego.  For  users,  this  meant  there  was  minimal  (or  no)  introduction  to  CommandNetis 
capabilities  and  very  little  training  on  the  tool,  which  clearly  affected  tool  usage. 


Another  important  factor  for  successful  transition  is  that  the  required  infrastructure  for  the  tool  be  known 
and  in  place  in  the  operational  organization.  Like  all  collaborative  applications,  CommandNet  interacts 
with  network  communications  to  support  real-time  response  to  users,  and  the  requirements  for  sufficient 
tool-communications  interaction  could  not  always  be  met  during  the  exercise.  This  affected  user  response 
and  ultimate  user  satisfaction  with  the  tool.  Another  aspect  of  infrastructure  is  security  policy,  and 
CommandNet  will  have  to  be  hardened  in  that  area  because  of  vulnerabilities  in  its  current  system 
administration  design.  Users  with  admin  privileges  appear  to  have  administrative  rights  over  all  logs, 
inconsistent  with  need  for  protection  of  enclaves  or  communities  of  interest.  Also,  editing  capabilities  are 
embedded  within  administrative  rights,  such  that  users  who  have  editing  capabilities  may  inadvertently  also 
have  administrative  privileges  that  were  not  explicitly  granted. 


With  respect  to  effective  evaluation  during  transition,  it  would  be  useful  for  versions  of  the  tool  to  be 
staged  in  a  systematic  way  so  that  appropriate  instrumentation  for  usage  data  collection  can  be  specified.  If 
features  of  the  system  evolve  and  are  fielded  too  rapidly,  the  user  and  the  evaluator  cannot  form  a  good 
model  of  the  system. 


Evaluation  Methodology 

This  research  project  improved  our  evaluation  methodology  by  illustrating  the  value  of  complementary 
perspectives  in  the  areas  of  data  collection,  analysis,  and  organizational  support. 


Combining  the  quantitative  data  with  the  qualitative  observations  created  a  veritable  treasure  chest  of 
useful  information.  The  two  types  of  data  complemented  each  other  and  provided  us  with  a  much  richer 
data  set,  giving  us  an  increased  degree  of  certainty  in  our  conclusions. 


Multiple  levels  of  analysis  contributed  to  a  more  complete  evaluation.  We  performed  a  traditional  HCI 
usability  study  on  the  collaborative  tool  (see  appendix  C).  We  also  examined  low-  and  mid-level  usage 
patterns  of  user  interactions  with  the  tool.  Finally,  we  examined  the  use  of  the  tool  from  a  social  and 
organizational  perspective. 
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Another  important  contribution  to  the  evaluation  effort  included  the  combined  input,  support,  and 
perspective  of  each  of  the  groups  involved:  MITRE,  CMI  and  the  Third  Fleet.  Together,  these  three 
organizations  created  a  strong  collaborative  research  environment. 


Conclusion:  Collaborative  Technology  Research  Perspective 


The  research  methodology  and  results  are  consistent  with  current  themes  in  the  development  of 
collaborative  technologies,  including 

•  importance  of  alignment  of  technologies  with  task/mission  being  performed; 

•  cultural  factors  that  affect  utility  and  adoption  of  technologies; 

•  accommodation  of  variation  in  user  behavior  within  and  across  tasks  and  user  communities;  and 

•  critical  need  for  evaluation  within  a  mission  context,  and  for  instrumentation  to  support  automated  data 
collection. 


Alignment  of  Technologies  and  Tasks 

Not  only  was  CommandNet  very  easy  to  use,  but  it  directly  supported  a  function  that  is  structurally  integral 
to  naval  operations:  the  keeping  of  logs  and  situation  reports.  There  was  general  agreement  about  what 
constituted  such  logs  and  sitreps  and  with  whom  they  were  shared.  Military  users  were  already  familiar 
with  the  process  of  log  entry  and  maintenance,  and  a  few  even  had  prior  experience  with  GroupSystems, 
the  predecessor  to  CommandNet.  Some  non-military  representatives  were  also  accustomed  to  making 
regular  tracking  reports.  For  such  users,  there  was  virtually  no  barrier  between  what  they  needed  to  do  and 
the  tools  they  had  for  performing  their  jobs. 


However,  use  of  CommandNet  was  not  limited  to  logging,  and  its  capabilities  were  explored  in  free  play. 
Users  recognized  its  broader  communication  function,  and  a  few  initially  exploited  it  as  a  ‘chat’  tool  until 
that  was  officially  discouraged.  It  is  interesting  to  note  that  later  its  utility  as  a  more  general 
communications  medium  became  both  visible  and  valuable  when  radio  connections  failed  between  the 
Coronado  and  the  command  center  ashore,  and  CommandNet  became  the  only  electronic  way  of 
exchanging  messages. 


CommandNet  was  also  seen  as  a  useful  data  repository,  and  users  expressed  interest  in  having  more 
complex  data  structuring  and  data  management  functions  as  additional  features.  (It  is  not  clear  how  well 
they  understood  that  existing  features  such  as  Category  could  have  met  some  of  their  needs.) 


The  point  is  that  users  may  begin  to  use  a  tool  for  specific  purposes,  but  can  evolve  their  usage,  and  also 
their  requirements,  into  areas  not  originally  within  the  design  center  of  the  tool. 


Cultural  Factors 

It  is  common  knowledge  in  the  area  of  collaboration  technologies  that  successful  transition  of  technologies 
into  operational  environments  requires  organizational  commitment  (to  collaboration,  to  the  tool,  and  to 
evaluation),  as  well  as  clear  requirements  and  good  alignment  between  the  technologies  and  tasks  to  be 
performed.  From  this  research,  it  appears  that  military  involvement  in  HA/DR  exercises  is  an  excellent 
testbed  for  evaluation  and  transition  of  a  collaborative  log  tool.  The  requirement  to  collaborate  across  time 
and  place  is  clear.  Missions  are  well-defined,  participants  are  well  trained,  they  share  doctrine  and 
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common  understanding  of  mission  components,  and  they  are  highly  motivated  to  use  whatever  means  are 
available  to  them  to  succeed. 


Variation  in  Usage 

As  mentioned  in  previous  sections,  there  was  considerable  variation  in  the  usage  of  the  tool  and  its 
perceived  utility.  There  were  eight  different  logs  used  during  these  exercises,  and  those  eight  logs 
illustrated  eight  different  usage  patterns.  This  is  not  surprising,  given  the  user  communities  and  the 
dynamic  context  in  which  they  were  operating.  A  principal  constraint  on  usage  and  learning  during  this 
exercise  was  compressed  time.  Exercise  events  occurred  at  a  pace  that  precluded  time  and  resources  for 
training  and  exploration. 


This  highlights  the  need  for  longitudinal  analysis  of  collaborative  interaction  and  how  dynamics  evolve 
over  time,  as  users  become  more  familiar  with  collaborative  tools  and  use  them  in  broader  aspects  of  their 
work  environment.  The  current  study  is  a  snapshot  and  what  may  really  be  needed  is  a  movie,  although  the 
data  collection  requirements  would  be  staggering. 


Evaluation  Issues 

Evaluation  is  as  inherent  a  part  of  tool  and  technology  development  as  writing  code.  Without  an  evaluation 
framework  and  real  data,  there  is  no  principled  way  to  determine  what  tools  are  appropriate  for  which  tasks, 
or  even  if  evolving  versions  of  a  tool  are  actually  ‘improvements.’  This  study  exemplifies  the  evaluation 
issues.  It  is  critical  to  evaluate  tools/technologies  in  contexts  that  are  meaningful  to  participants  and  that 
require  collaboration  for  effective  task/mission  performance.  Instrumented  data  capture,  interviews, 
questionnaires,  and  ongoing  observation  are  all  essential  to  gaining  a  full  understanding  of  a  tool’s 
effectiveness.  Analysis  must  reflect  both  the  patterns  from  quantitative  data  and  the  texture  of  the  social 
interactions  that  underlie  collaboration. 
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Feedback  on  CommandNet  Date:  __l^. 

Please  take  a  few  minutes  to  fill  out  this  survey.  Your  input  is  greatly  appreciated. 


Background  Information 


1.  Gender:  M _  F 


2.  Age  Group: 

under  20  20-29  30-39  40-49  50+ 


3.  Rank  and  specialization? 

4.  On  average  how  many  hours  per  week  do  you  spend  using  a  computer? 
0-1  1-10  10+ 


5.  Check  all  the  computer-supported  collaboration  tools  that  you  have  used. 

GroupSystems _  email _  chat _  whiteboard _ 

shared  directories _  audio _  video _  document  management  software _ 

other  (please  specify) _ 

6.  Before  using  CommandNet,  what  media  did  you  use  to  report  your  observations,  and 
to  access  information?  (e.g.  message  traffic,  notebook,  etc.) 


Feedback  on  the  features 


7.  Rate  the  category  feature: 

Useful  1  2  3  4  5  Useless  _ Did  not  use 

Easy  1  2  3  4  5  Difficult 

Comments  on  the  category  feature: 


8.  Rate  the  importance  feature: 

Useful  1  2  3  4  5  Useless  _ Did  not  use 

Easy  1  2  3  4  5  Difficult 

Comments  on  the  importance  feature: 


9. 


Rate  the  search  feature: 


Useful 

Helpful 

Easy 


1 

2 

3 

4 

5 

Useless 

Did  not  use 

1 

2 

3 

4 

5 

Hindrance 

1 

2 

3 

4 

5 

Difficult 

Comments  on  the  search  feature: 
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10.  Rate  the  headlines  feature: 


Useful 

Helpful 

Easy 


1 

2 

3 

4 

5 

Useless 

Did  not  use 

1 

2 

3 

4 

5 

Hindrance 

1 

2 

3 

4 

5 

Difficult 

Comments  on  the  headlines  feature: 


Overall  Experience 


1 1 .  Rate  the  training  you  received  in  CommandNet: 

Adequate  1  2  3  4  5  Inadequate  _ No  training 

12.  How  would  you  compare  using  CommandNet  with  the  method(s)  you  have  used  in 
the  past?  Using  CommandNet  is 


Easier 

1 

2  3  4  5 

Harder 

No  opinion 

Faster 

1 

2  3  4  5 

Slower 

No  opinion 

More  fun 

1 

2  3  4  5 

More  irritating 

No  opinion 

13.  Analyzing  logs  using  CommandNet: 

Is  faster  1  2  3  4  5  Slower  than  previous 

No  opinion 

Provides  better 

1 

2  3  4  5 

method(s) 

Worse  quality  analyses 

No  opinion 

14.  Rate  the  overall  experience  of  using  CommandNet: 

Useful  1  2  3  4  5  Useless 

No  opinion 

Appropriate 

1 

2  3  4  5 

Inappropriate  for  my 

No  opinion 

Easy 

1 

2  3  4  5 

tasks 

Difficult 

No  opinion 

15.  What  did  you  like  most  about  CommandNet?  What  things  were  easy? 


1 6.  What  did  you  like  least?  What  did  you  have  the  most  trouble  with? 


17.  Was  there  something  you  wanted  to  do  in  CommandNet  but  could  not  do? 
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Appendix  B:  Detailed  Results 

This  appendix  details  our  data  findings  of  CommandNet  usage  during  Strong  Angel  and  RIMPAC  2000  exercises. 
The  data  discussed  in  this  appendix  were  taken  from  server  logs,  content  logs,  questionnaires,  and  interviews.  Tool 
usage  differed  greatly  between  the  humanitarian  assistance  /  disaster  relief  exercise  of  Strong  Angel  and  the  naval 
exercises  of  RIMPAC  2000,  and  so  we  keep  the  discussions  separate. 

We  start  with  an  overview  of  the  logs  used  in  the  exercises  and  a  user  study.  Then  we  examine  the  use  of  the  Basic 
Features,  making  entries,  and  the  associated  use  of  categories  and  importance.  Following  that,  we  take  a  look  at 
more 

Advanced  Features  such  as  search,  headlines,  preferences,  and  hyperlinks.  We  include  a 
Miscellaneous  subsection  to  investigate  other  observations.  Finally,  we  profile  each  user  community. 


Overview  of  Logs  and  Users 

This  subsection  provides  an  overview  of  the  logs  and  user  accounts  created  and  used  in  both  Strong  Angel  and 
RIMPAC  2000.  Also  included  is  a  study  on  logging  in  and  logging  out. 

Logs 

There  were  three  logs  used  during  Strong  Angel  on  the  Big  Island  and  five  used  during  RIMPAC  2000.  There  were 
additional  logs  used  for  testing  purposes  plus  a  few  logs  created  but  never  used.  Table  1  lists  all  created  logs. 


Exercise 

Log  Name 

Description 

Strong  Angel 

CMOC 

Civil  Military  Operations  Center  Log 

CMOC  Afloat 

JOC  Log 

Civil  Affairs 

Civil  Affairs  Log 

Test  Log 

Test  log 

New  Test  Log 

Test  log  for  demo  purposes 

RIMPAC  2000 

BWC 

Battle  Watch  Captain’s  Log 

JCCC 

Joint  Command  and  Control  Center  (technical 
support) 

Intel 

C3F  Intelligence  Log 

CFACC 

Coalition  Forces  Air  Command  Control  Log 

CECG 

Coalition  Exercise  Control  Group  Log 

ARFOR 

Unknown  (little  used) 

Test  Log 

Test  log 

Table  1  CommandNet  logs  created  for  Strong  Angel  and  RIMPAC  2000. 


Strong  Angel 

There  were  three  main  logs  used  in  Strong  Angel:  CMOC,  CMOC  Afloat,  and  Civil  Affairs.  (We  exclude  all  data 
from  the  two  test  logs  in  all  our  analyses.) 

The  CMOC  log  was  the  original  log  created  for  three  groups:  C3F  at  sea,  UN  agencies  on  land,  and  USMC  at  the 
CMOC  and  refugee  camps.  The  CMOC  Afloat  log  was  created  when  a  Third  Fleet  officer  requested  a  log  that  was 
not  shared  with  the  UN  or  the  USMC.  (This  log  contains  some  entries  duplicated  from  the  CMOC  log.)  The  Civil 
Affairs  log  was  created  to  filter  information  from  the  refugee  camp  before  it  went  into  the  CMOC  log.  It  also 
supported  Civil  Affairs  communication  more  generally. 

Figure  1,  below,  shows  how  many  participants  used  each  log.  The  more  heavily-used  log,  the  CMOC  log,  was  used 
by  18  participants.  Both  the  CMOC  Afloat  and  Civil  Affairs  logs  had  8  users.  Note  that  these  counts  are  not 
entirely  accurate  and  may  be  deflated;  it  was  observed  that  a  number  of  participants  made  entries  under  another 
user’s  account. 
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Users  per  Log  in  Strong  Angel  ' 


Figure  1  Number  of  users  in  Strong  Angei  iogs. 


IRIMPAC2000 

There  were  five  main  logs  used  during  RIMPAC  2000.  The  ARFOR  log  was  created  and  used  minimally  around  its 
time  of  creation.  It  was  never  used  again.  Except  for  this  subsection,  we  exclude  all  data  from  both  the  ARFOR  log 
and  the  test  log  in  our  analysis  and  discussion. 

Figure  2  shows  the  number  of  users  in  each  log.  The  BWC  Log  had  the  largest  number  of  users  by  a  significant 
amount.  Note  that  this  graph  can  be  deceptive  since  some  logs  were  used  by  multiple  people  using  a  single  account, 
or  role.  As  discussed  in  a  later  subsection,  all  entries  in  the  CECG  log  were  made  by  one  account,  the  CECG  watch 
officer’s  account.  This  one  account,  or  role,  was  used  by  multiple  participants. 


BWC  JCCC  Intel  CFACC  CECG  ARFOR  Test 


Figure  2  Number  of  users  in  RIMPAC  2000  logs. 


Users 

We  approximate  the  number  of  participants  using  CommandNet  during  the  exercises.  The  actual  number  of  people 
using  CommandNet  and  the  number  of  users  may  differ  considerably  since  multiple  people  often  used  the  same 
login  account  (multi-user  role  accounts)  or  made  entries  under  someone  else’s  login  account. 

Most  users  had  access  to  one  or  two  logs  at  most.  A  few  had  access  to  all  available  logs.  People  who  had  access  to 
all  logs  were  usually  administrative  people. 
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[Strong  Angel 

There  were  23  user  accounts  created  for  Strong  Angel  logs.  20  of  those  accounts  were  actually  used.  Only  one 
account  was  created  for  a  “role.”  All  other  accounts  were  created  for  individuals.  It  was  observed  that  one  of  the 
individual  accounts  was  actually  used  by  multiple  participants.  From  the  data,  it  is  unknown  if  other  users  did  not 
have  their  own  accounts  or  if  no  one  ever  bothered  to  log  out. 

50%  of  those  users  accessed  2  logs,  40%  accessed  just  a  single  log.  Only  10%  had  access  to  all  3  logs,  and  those 
were  probably  CMI  members  and  the  MITRE  person.  Access  to  logs  was  determined  by  permissions  set  for  each 
user  account.  Figure  3  illustrates  the  relationship  between  the  number  of  users  and  the  number  of  logs  the  users 
accessed. 


Logs  per  User  in  Strong  Angel  ' 


Figure  3  Accessed  logs  per  user  in  Strong  Angel.  Some  users  had  access  to  two  logs,  while  almost  as  many 
users  had  access  to  just  a  single  log.  Very  few  could  access  all  three. 


[RIMPAC  2000 

There  were  332  accounts  created  for  use  in  RIMPAC  2000.  Of  those,  some  were  roles  used  by  single  users,  and 
some  were  roles  used  by  multiple  users.  Of  those  332  accounts,  only  92  were  actually  used,  and  that  included 
administrators  and  members  of  the  University  of  Arizona’ s  Center  for  the  Management  of  Information  team  plus  the 
MITRE  observer. 

Many  users  who  did  not  use  their  own  individual  accounts,  used  CommandNet  under  single-user  or  multi-user  roles. 


Type  of  Account  in  RIMPAC  2000 
Logs 

#  Accounts 
Created 

#  Accounts  Used 

CMI 

4 

3 

MITRE 

1 

1 

single-user  roles 

89 

14 

multi-user  roles 

37 

13 

Individual 

201 

59 

Total 

332 

90 

Table  2  Types  of  accounts  created  and  used  in  RIMPAC  2000. 
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50%  of  users  accessed  just  a  single  log  in  RIMPAC  2000  while  36%  accessed  2  logs.  Only  1%  accessed  either  5 
logs  or  all  7. 


Logs  per  User  in  RIMPAC 


#  Logs 


Figure  4  Accessed  logs  per  user  in  RIMPAC  2000.  While  the  majority  of  users  had  access  to  just  a  single  log, 
a  substantial  number  of  users  could  access  two  logs.  Very  few  could  access  three  or  more. 

Login  /  Logout _ 

There  was  no  automatic  data  capture  of  login  into  CommandNet.  The  closest  approximation  to  logging  in  was  taken 
from  the  number  of  times  a  user  chose  a  log.  Users  could  chose  a  log  upon  login  or  after  opening  up  another  log. 

We  approximated  login  numbers  by  the  number  of  times  the  user  went  to  the  Choose  Log  screen  when  no  other  log 
had  previously  been  chosen. 

A  logout  button  was  available  in  the  menu  at  the  top  of  each  window.  Alternatively,  a  user  could  close  the  browser 
window  or  exit  the  browser  without  logging  out.  In  fact,  the  ratio  of  logging  out  to  logging  in  was  very  low.  It 
appeared  that  very  few  users  actually  logged  out  on  a  regular  basis  (see  Table  3). 


Frequency  of  Logging  Out 

Strong  Angel 

RIMPAC  2000 

Logged  out  every  time 

11% 

6% 

Logged  out  most  of  the  time  (-75%) 

11% 

0% 

Logged  out  half  of  the  time  (-50%) 

16% 

4% 

Logged  out  infrequently  (1-50%) 

26% 

31% 

Never  logged  out 

37% 

58% 

Table  3  Frequency  of  logging  out  of  CommandNet. 


[Strong  Angel  I 

In  Strong  Angel,  only  1 1%  of  the  users  actually  logged  out  every  time.  1 1%  of  the  users  logged  out  most  of  the  time 
(75%).  16%  of  the  users  logged  out  half  the  time.  25%  of  the  users  logged  out  infrequently  (1-50%  of  the  time). 

37%  never  logged  out  at  all.  Overall,  the  logout  rate  was  27%. 


IRIMPAC  2000 

Users  of  CommandNet  in  RIMPAC  2000  logged  out  much  less  frequently  than  users  during  Strong  Angel.  The 
RIMPAC  logout  rate  was  a  low  6%.  Only  6%  logged  out  every  time,  and  over  half  the  users  (58%)  never  logged 
out  at  all,  while  a  full  31%  logged  out  infrequently  (less  than  half  the  time). 


Figure  5,  below,  indicates  that  the  BWC  log  and  the  Intel  log  were  opened  more  than  the  other  logs.  The  BWC  log 
was  opened  about  42%  of  the  time,  the  Intel  log  about  30%  of  the  time.  The  CFACC  and  CECG  logs  were  opened 
less  than  6%  of  the  time.  However,  this  could  mean  that  the  logs  stayed  open  longer,  not  necessarily  that  they  were 
used  less.  This  graph  closely  resembles  the  one  in  Figure  2  Number  of  users  in  RIMPAC  2000  logs.  In  general,  the 
number  of  users  per  log  is  proportional  to  the  number  of  times  a  log  was  opened.  The  one  difference  is  apparent  in 
the  CFACC  log  which  was  probably  closed  infrequently. 
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Likelihood  of  Opening  a  Given  Log 


Figure  5  Likelihood  of  Opening  a  Given  Log  in  RIMPAC  2000.  This  graph  shows  which  logs  were  opened 
most  often. 


Logout  vs  Open  Log 


BWC  JCCC  Intel  CFACC  CECG 


Figure  6,  at  left,  shows  that  users  were 
more  apt  to  logout  of  the  CFACC  log 
than  any  other  log.  Since  the  CFACC 
log  was  opened  so  few  times,  this 
indicates  that  users  of  the  CFACC  log 
were  better  at  logging  out  with  the 
logout  button. 


Figure  6  Logout  versus  Open  Log  in  RIMPAC  2000.  Users  who  opened  the  CFACC  log  first  were  better  at 
logging  ou  t  than  users  of  other  logs. 


Likelihood  of  Logging  Out  from  Given  Log 


The  last  graph  in  this  series.  Figure  7, 
shows  the  likelihood  of  logging  out 
from  a  specific  log.  The  numbers  are 
all  small,  but  more  people  logged  out 
of  the  BWC  log  than  any  other  log. 
The  null  column  indicates  that  a 
number  of  users  logged  out  without 
ever  choosing  a  log.  In  other  words, 
they  logged  out  just  after  logging  in. 


Figure  7  Likelihood  of  Logging  Out  from  a  Given  Log. 
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Basic  Features 


Entries _ 

We  examined  the  frequency  of  entries  made  in  CommandNet  and  the  wordiness  of  those  entries  for  changes  in 
patterns  over  time.  We  also  studied  the  content  of  entries,  but  that  is  discussed  later  in  the  Categories  subsection. 


IStrong  Angel 

CommandNet  was  in  use  for  just  seven  days.  Entries  were  mostly  made  on  just  four  of  those  days.  Figure  8 
illustrates  the  pattern  and  number  of  entries  made  in  CommandNet  across  all  logs.  Fewer  than  40  entries  were  made 
per  day.  Figure  9  graphs  the  average  number  of  words  used  in  entries  over  time.  On  the  12*,  entries  were  copied 
from  one  log  to  another. 


Entries  per  day  during  Strong  Angei 
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Figure  8  Frequency  of  entries  in  CommandNet  per  day  during  Strong  Angel. 
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Figure  9  Average  words  per  entry  per  day  during  Strong  Angel. 
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entries,  over 
Figure  10 
over  time  by 

log. 


Table  4  below  contains  statistics  on  the  entries  made  in  all  of  the  logs.  The  CMOC  log  had  the  most 
half  the  total  entries,  but  the  Civil  Affair  entries  were  longer,  with  an  average  of  45  words  per  entry, 
shows  the  frequency  of  entries  over  time  by  log  while  Figure  1 1  shows  the  average  number  of  words 


CMOC 

CMOC  Afloat 

Civil  Affairs 

Total 

total  entries 

71 

23 

21 

115 

#  blanks 

0 

0 

0 

0 

#  duplicates 

0 

0 

0 

0 

#  tests 

8 

2 

0 

10 

clean  total  (without  blanks, 
duplicates,  tests) 

63 

21 

21 

105 

average  words  per  entry 

27 

33 

45 

Table  4  Statistics  on  entries  made  in  Strong  Angei  iogs. 


Entries  per  day  per  log  during  Strong  Angel  " 


140 

120 

100 

oo 

.22  80 
L_ 

-I— ■ 

It  60 
40 

20 


11  13  15  17  19  21  23  25  27 

Day 


Figure  10  Entries  per  day  per  iog  during  Strong  Angei.  On  the  12*^,  the  CMOC  Afioat  log  was  created,  and  12 
entries  were  dupiicated  from  the  CMOC  iog.  The  Civil  Affairs  iog  was  created  on  the  14^. 
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Figure  11  Average  words  per  entry  per  day  during  Strong  Angei.  The  spike  in  the  CMOC  Afioat  iog  occurs 
where  the  partial  contents  of  a  document  were  cut  and  pasted  into  the  log. 


IRIMPAC2000 

RIMPAC  logs  were  used  considerably  more  than  Strong  Angel  logs.  They  contained  a  total  of  2358  entries  over  23 
days,  excluding  entries  made  to  the  Test  Log.  25  entries  were  tests.  5  entries  were  blanks,  probably  submitted  in 
error.  71  of  those  entries  were  duplicates,  or  3%  of  all  entries.  Interviews  with  users  showed  considerable 
complaint  that  the  system  was  running  so  slowly  that  entries  were  often  submitted  more  than  once  because  they  did 
not  appear  in  the  logs  within  a  reasonable  time  period.  The  presence  of  duplicate  and  triplicate  entries  supports  this. 


BWC 

JCCC 

Intel 

CFACC 

CECG 

ARFOR 

Total 

total  entries 

555 

608 

705 

314 

174 

2 

2358 

#  blanks 

1 

1 

0 

3 

0 

0 

5 

#  duplicates  (%  of  clean 
total) 

15 

(3%) 

27 

(5%) 

17 

(2%) 

8  (3%) 

4  (2%) 

0 

71  (3%) 

#  tests 

6 

3 

3 

5 

6 

2 

25 

clean  total  (without 
blanks,  duplicates,  tests) 

534 

584 

685 

301 

168 

0 

2257 

average  words  per  entry 

22.3 

16.3 

18.2 

18.5 

24.4 

0 

19.2 

median  words  per  entry 

15 

13 

13 

12 

14 

0 

13 

min  words  per  entry 

1 

3 

3 

1 

3 

0 

1 

max  words  per  entry 

305 

257 

217 

422 

229 

0 

422 

Table  5  Statistics  on  entries  made  in  RIMPAC  2000  logs. 


On  average,  each  entry  was  about  19  words  in  length,  shorter  than  entries  in  Strong  Angel  logs.  Entries  made  to  the 
JCCC  log,  the  technical  support  log,  were  the  shortest. 

and  Figure  13  show  entries  and  entry  length  over  time.  The  BWC,  Intel,  and  JCCC  logs  were  used  throughout  the 
entire  month.  The  CFACC  and  CECG  logs  were  not  used  until  the  RIMPAC  exercises.  Activity  in  all  logs 
drastically  increased  after  the  20^^  when  exercise  activity  increased. 
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Figure  12  Frequency  of  entries  in  RiMPAC  2000.  On  the  19*^,  there  were  oniy  three  entries,  aii  in  the  intei  iog. 
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Figure  13  Average  words  per  entry  per  day  in  RiMPAC  2000  iogs.  The  3  intei  entries  on  the  19*^  were  iengthy. 


The  same  data  are  separated  by  log  in  the  graphs  shown  in  Figure  14  and  Figure  15.  It  is  clear  that  the  Intel  log  was 
used  very  heavily  during  RIMPAC  exercises.  The  CFACC  and  CECG  logs  were  used  moderately  during  the 
exercises.  The  JCCC  log  was  used  consistently  throughout  the  monitored  period.  Usage  of  the  BWC  log  was 
somewhat  consistent  but  ramped  up  during  the  exercises. 
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Day 


Figure  14  Entries  per  day  per  iog  in  RiMPAC  2000  iogs.  Activity  in  the  intei  iog  increases  considerabiy  after 
the  20*^  when  RiMPAC  2000  exercises  are  underway.  Activity  in  the  BWC  iog  aiso  ramps  up  after  the  20*^. 
The  CFACC  and  CECG  iogs  are  not  used  at  aii  untii  that  point. 


Day 


Figure  15  Average  words  per  entry  per  day  per  iog  in  RiMPAC  2000  iogs.  The  JCCC  iogs  are  the  most 
consistent  in  length,  around  16  words  per  entry.  The  length  of  the  entries  does  not  change  much  even  after 
the  20*^  when  exercise  activity  increases. 


The  frequency  of  duplicate  entries  follows  a  pattern  similar  to  the  frequency  of  entries  made.  (See  Figure  16, 
below.)  Most  duplicate  entries  occurred  during  periods  when  exercise  activity  was  most  intense.  Users  reported 
that  log  updating  was  so  slow  that  they  hit  the  submit  button  twice  or  more. 
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Figure  16  Frequency  of  Duplicate  Entries  in  RIMPAC  2000  logs. 
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Figure  17  Frequency  of  Duplicate  Entries  by  Log.  The  activity  peak  after  the  20*^  shows  that  duplicates  were 
made  in  all  logs  by  the  CECG  log.  Prior  to  that  time  period,  most  duplicates  occurred  in  the  JCCC  log  where 
the  frequency  of  duplicates  appeared  to  be  consistent  throughout  the  entire  month. 


Categories 

The  category  feature  was  an  easily-accessible  feature  to  creating  entries.  Before  an  entry  was  submitted,  the  user 
could  select  a  label,  or  category,  from  a  dropdown  list.  The  default  category  was  “None”. 

Categories  provided  an  easy  way  to  classify  entries  by  type.  The  more  advanced  Search  feature  also  provided  a 
means  for  searching  by  category. 


[Strong  Angel 

There  were  10  categories  created  for  use  in  the  Strong  Angel  logs.  All  10  were  created  by  CMI  members. 
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Table  6  lists  the  categories  created  for  use  in  the  Strong  Angel  logs  and  how  much  each  category  was  actually  used. 
Categories  were  not  used  at  all  in  the  CMOC  Afloat  Log.  Entries  in  CMOC  log  used  categories  just  53%  of  the 
time,  whereas  the  Civil  Affairs  log  used  categories  91%  of  the  time. 


Strong  Angel  Log 

Category 

#  Entries 

CMOC 

PW  Base 

not  used 

High  Security 

not  used 

Information 

14 

Dispatch 

5 

Lesson 

3 

Civil  Affairs 

18 

SITREP 

1 

None 

34 

Civil  Affairs 

Base  Camp 

7 

Refugee  Camp 

14 

None 

2 

Table  6  Categories  usage  in  Strong  Angei  iogs. 


Why  were  categories  used  fairly  consistently  in  one  log,  about  half  the  time  in  another  log,  and  never  in  the  third 
log?  One  possible  explanation  has  to  do  with  learning  by  example.  If  a  new  user  saw  existing  entries  with 
associated  categories,  he  may  have  been  more  likely  to  create  entries  with  categories.  Likewise,  if  visible  entries  in 
the  log  had  no  categories  associated  with  them,  a  new  user  would  be  less  likely  to  create  and  entry  with  a  category. 

An  explanation  for  the  total  lack  of  category  usage  in  the  CMOC  Afloat  log  could  be  that  no  categories  were 
available.  Similarly,  perhaps  the  categories  defined  in  the  CMOC  log  were  not  appropriate  for  the  type  of  entry 
made.  Users  added  categories  to  entries  when  there  was  a  match  and  did  not  use  categories  when  there  was  not  a 
good  fit. 

We  pursued  this  line  of  thinking  and  examined  the  actual  entries.  We  created  our  own  list  of  categories  that  we  call 
“  content  types.”  We  then  annotated  each  entry  in  Strong  Angel  with  these  content  types.  Note  that  an  entry  can 
contain  more  than  one  content  type.  For  example,  a  SITREP  entry  often  contained  tracking  information. 

Definitions  of  our  content  types  and  their  frequency  appear  in  Table  7. 


Content  Type 

Definition 

%  Entries 

sitrep 

situation  report 

47% 

tracking 

personnel,  resources,  refugees,  food,  etc 

35% 

trouble  shooting 

report,  update,  or  solution  of  a  problem,  maintenance,  etc 

16% 

informal  chat 

chat  not  related  to  any  of  the  other  categories 

14% 

request  for  action 

11% 

acknowledgment 

acknowledgment  of  a  request  or  that  a  message  was  read 

10% 

test 

any  type  of  test  entry 

10% 

request  for  info 

5% 

request  for  sitrep 

4% 

scheduling 

proposed  schedule  or  agenda 

3% 

report  /  document 

a  formal  report,  article,  or  document 

3% 

correction 

amendment  to  prior  entry 

1% 

system 

any  reference  to  CommandNet 

1% 

Table  7  Content  types  and  frequency  in  Strong  Angel  logs. 
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Figure  18  Content  types  of  entries  in  Strong  Angei  iogs. 
reports  and  tracking. 


The  usage  of  these  content  types  is  graphed  in  Figure 
18.  It  is  clear  from  the  data  that  CommandNet  was 
used  most  to  report  sitreps  (47%)  followed  by  tracking 
information  (35%).  There  was  significant  use  for 
troubleshooting,  chat,  request  for  action, 
acknowledgment,  and  testing  (all  >  10%). 


The  most  common  entry  types  were  situation 


How  does  the  use  of  categories  compare  to  our  annotated  content  types'? 


The  following  series  of  graphs  shows  the 
juxtaposition  of  categories  and 
content  types.  Each  graph 
represents  one  category  and 
shows  where  it  overlapps  with 
our  annotated  types. 

‘Base  Camp’  categorized  entries 
atypically  contained  more 
tracking  entries  than  sitrep 
entries.  Base  Camp  entries  were 
also  used  for  troubleshooting, 
sitrep  requests,  and  over  20% 
acknowledgments . 


Figure  19  Content  types  of 
entries  that  were  categorized 
as  Base  Camp’ 


The  ‘Civil  Affairs’  category  (all  from  the  CMOC  log)  was  almost  exclusively  used  for  sitreps  and  tracking. 


Figure  20  Content  types  of 
entries  that  were  categorized 
as  ‘Civii  Affairs’ 
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Content  Types:  Information 


The  ‘Information’  category  was 
used  for  almost  all  of  our  content 
types. 


Figure  21  Content  types  of  entries  that  were  categorized  as  ‘Information’ 
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The  ‘Lesson’  category  was  used 
mostly  for  troubleshooting  (-65%) 
and  for  informal  chat  (-35%). 


Figure  22  Content  types  of  entries  categorized  as  ‘Lesson’ 


Figure  23  Content  types  of  entries  categorized  as  ‘Refugee  Camp’ 


The  ‘Refugee  Camp’  category 
was  used  in  a  similar  pattern  to 
the  use  of  ‘Civil  Affairs’:  mostly 
sitreps  and  tracking  in  the  Civil 
Affairs  log. 
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The  ‘SITREP’  category  was  used 
for  informal  chat  -  in  this  one 
entry,  it  was  a  reference  to  the 
location  of  sitrep  documents. 


Figure  24  Content  type  of  the  single  ‘SITREP’  category  entry 


Figure  25  Content  types  of  entries  categorized  as  ‘Dispatch’ 
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Content  Types:  no  category 
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Entries  that  were  not  categorized 
fell  into  almost  all  of  our  content 
types. 


Figure  26  Content  types  of  entries  with  no  category 
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IRIMPAC  2000 

In  RIMPAC  logs,  categories  were  used  when  creating  entries  almost  exclusively  in  the  Intel  log.  77%  of  Intel 
entries  used  categories.  There  were  10  entries  in  the  BWC  log  that  used  categories.  No  other  logs  had  categories 
associated  with  entries. 


RIMPAC  Log 

Category 

#  Entries 

BWC 

None 

533 

SITREP 

8 

MEDEVAC 

1 

Medical  Casualty 

1 

Intel 

None 

160 

Air  Defense 

146 

Air 

121 

TBMD 

85 

Naval-Surface 

65 

Coastal  Defense 

57 

Naval-Subsurface 

33 

Ground 

31 

Targeting 

1 

HADR 

0  -  not  used 

Table  8  Category  Usage  in  RIMPAC  2000  Logs.  The  default  is  ‘None’  -  no  category. 

Listed  below  are  potential  categories  that  were  not  used  in  CommandNet. 

ETR  estimated  time  to  repair 

OPSUM  operational  summary 

SORTS  operational  status  of  resources  and  training 

CASREP  casualty  report  (broken  equipment) 


oo 


tt 


Frequency  of  Category  Use 


Day 


The  graph  at  left  contrasts  the 
use  of  categories  between  the 
BWC  log  and  the  Intel  log. 
Entries  in  the  BWC  log  are 
shown  by  the  broken  blue  line. 
Categories  are  used 
infrequently,  at  best,  as  shown 
by  the  solid  blue  line.  By 
contrast,  the  use  of  categories 
(solid  green  line)  in  the  Intel  log 
remains  proportional  to  the 
regular  use  of  entries  (broken 
green  line). 


Figure  27  Frequency  of  category  use  in  BWC  and  intei  logs 


Importance _ 

The  importance  feature  was  another  easily-accessible  label  to  data  entries.  The  default  importance  level  was 
‘Routine.’ 


Importance 

Display  Code 

Description 

Associated  Time 

Routine 

R 

Routine 

None 

Priority 

P 

Priority 

~  6  hours 

Immediate 

0 

Operational  immediate 

<  30  minutes 

Plash 

Z 

Plash 

<10  minutes 

Table  9  Importance  levels  and  their  dispiay  codes. 
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The  drop  down  box  for  choosing  importance  listed  the  levels  as  words  (see  the  first  column  in  Table  9).  However, 
the  importance  level  associated  with  a  created  entry  was  displayed  as  a  single-letter  code.  There  were  some 
comments,  in  the  interviews  and  surveys,  that  people  did  not  know  what  these  letters  meant  -  one  explanation  as  to 
why  importance  was  little  used. 

One  design  recommendation  is  to  include  these  importance  levels  in  the  help  pages,  together  with  the  associated 
times. 


[Strong  Angel 

Importance  was  not  used  much  during  Strong  Angel.  Flash  was  never  used  at  all.  Priority  was  used  for  5%  of  the 
CMOC  Afloat  entries  (actually,  it  was  just  one  entry)  and  1 1%  of  the  CMOC  entries.  Immediate  was  used  only  in 
the  CMOC  log,  6%  of  all  entries. 


Use  of  Importance  by  Log 


100% 

Qj  80% 

60% 

LU 

40% 

o 

S  20% 
0% 


■  Routine 

■  Priority 

■  Immediate 

■  Flash 


T -  - 1 -  - 1 


Civil  CMOC  CMOC 

Affairs  Afloat 


Figure  28  The  Use  of  Importance  in  Strong  Angel  logs.  Importance  was  not  used  in  the  Civil  Affairs  log. 
Flash  was  never  used. 


IRIMPAC  2000 

The  importance  feature  was  used  more  in  the  RIMPAC  logs,  but  mostly  in  just  the  Intel  and  CFACC  logs. 


Level  of  Importance  Used  in  Entries 

Log 

Routine 

Priority 

Immediate 

Flash 

BWC 

96.6% 

2% 

1.1% 

0.3% 

JCCC 

97% 

0.1% 

0.3% 

0% 

Intel 

17% 

30% 

42% 

11% 

CFACC 

66% 

9% 

5% 

20% 

CECG 

99% 

1% 

0% 

0% 

Table  10  The  use  of  the  importance  feature  in  RIMPAC  2000  logs 


Users  of  the  CFACC  log  commented  that  they  did  not  think  anyone  was  monitoring  or  reviewing  the  logs.  If  that 
was  the  case,  why  did  they  bother  to  add  importance  levels?  Perhaps  importance  levels  were  used  because  of  the 
precedence  set  in  the  first  entries.  In  Figure  32  reveals  that  the  first  entries  made  were  flash  entries. 

There  were  three  levels  of  Intel  users:  those  entering  data  coming  in  from  various  data  streams,  those  monitoring  the 
BWC  and  CFACC  logs  and  entering  data  (anchor  desks),  and  those  making  higher  level  analyses  based  on  the 
entries  of  the  other  users.  These  three  groups  of  users  within  the  Intel  community  were  aware  of  each  other.  This 
could  be  why  the  Intel  logs  used  importance  more  than  any  other  community  of  users. 
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Use  of  Importance 


BWC  JCCC  Intel  CFACC  CECG 


Figure  29  Use  of  the  Importance  Feature  by  Log  in  RIMPAC  2000 

The  next  graph,  Figure  30,  shows  the  use  of  importance  by  log  as  a  percentage  of  all  entries  made.  This  shows  that 
Intel  users  made  a  large  percentage  (-40%)  of  Immediate  entries,  and  CFACC  users  made  about  20%  Flash  entries. 


Use  of  Importance  by  Log 


BWC  JCCC  Intel  CFACC  CECG 


Figure  30  Percentage  of  Importance  Use  by  Log  in  RIMPAC  2000 


The  following  series  of  five  graphs  show  the  frequency  of  importance  use  per  log.  The  most  interesting  graphs  are 

the  Intel  log  and  the 
CFACC  log. 


The  pattern  of  usage 
of  importance  in  the 
Intel  log  follows  the 
Intel  entry 
distribution.  The 
percentages  of 
priority,  immediate, 
and  flash  entries 
increase  as  the 
number  of  entries 
increase. 


Figure  31  Frequency  of  Importance  Use  in  the  Intel  Log 
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Figure  33 
Frequency  of 
Importance  Use  in 
the  BWC  Log 
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Figure  34 
Frequency  of 
Importance  Use  in 
the  JCCC  Log 
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Search 


We  have  thrown  out  all  searches  done  by  CMI  or  MITRE  staff,  and  any  searches  performed  on  test  logs. 


iStrong  Anflel 

In  the  Strong  Angel  logs,  only  2  users  (1%  of  active  users)  navigated  to  the  search  screen.  Both  of  these  users 
performed  a  search.  Only  one  of  the  users  actually  saved  the  search.  That  same  user  viewed  the  contents  of  the 
saved  search.  These  two  users  were  the  same  two  Strong  Angel  users  who  also  looked  at  headlines.  (See  below.) 


IRIMPAC2000 

In  RIMPAC,  22%  of  the  users  looked  at  the  search  page.  80-90%  of  those  users  never  went  back  to  the  search  page. 

(2  were  multi-user  roles,  and  so  probably  represented  more  than  one  user.) 

30%  of  those  users  made  a  search,  mostly  the  first  time  they  looked  at  the  search  page.  2  users  searched  once,  2 

searched  twice,  one  searched  4  times,  and  one  searched  1 1  times  on  2  different  days. 

Just  one  user  saved  a  search.  No  one  viewed  the  saved  search  (search  was  saved  on  the  26^^,  late  in  the  exercise). 

Profile  of  two  Intel  users  who  went  to  the  search  page  more  than  once: 

•  u321  went  to  the  search  page  3  times  on  one  day  and  did  1 1  searches,  three  times  two  days  later  without 
searching. 

•  u49  went  to  the  search  page  on  one  day  and  did  nothing.  The  next  day,  u49  went  back  to  the  search  page  and 
made  a  search.  Later  that  day,  u49  went  back  to  the  search  page  and  made  three  more  searches.  The  last  search 
was  saved.  It  was  never  viewed. 


Headlines _ 

This  Study  excludes  all  activity  done  by  CMI  or  MITRE  staff  and  any  activity  done  on  the  test  logs. 


[Strong  Angel 

2  Strong  Angel  users  (1%  of  active  users)  navigated  to  the  headlines  screen.  One  of  those  two  looked  at  headlines  in 
two  different  logs.  Both  users  tried  to  create  new  headlines,  both  added  entries  to  the  headlines.  Neither  user  saved 
the  headline.  These  are  the  same  two  Strong  Angel  users  who  did  searches.  (See  above.) 


IRIMPAC2000 

28  RIMPAC  users  (30%  of  active  users)  looked  at  the  headlines  page  at  least  once.  Of  those  28,  only  5  (5.5%  of  all 
users)  looked  more  than  once  on,  at  most,  two  separate  occasions. 
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5  of  those  28  tried  to  create  a  new  headline.  Only  one  of  those  5  tried  twice.  2  of  those  5  added  entries  to  the 
headline. 

Not  a  single  user  submitted  a  headline.  No  users  viewed  any  headlines  (there  was  none  to  view). 


Refresh _ 

CommandNet  was  set  to  refresh  every  60  seconds,  by  default.  Refreshing  updated  the  window  with  the  latest 
entries,  searches,  and  headlines.  When  a  refresh  occurred,  the  list  (of  entries,  searches,  and  headlines)  would 
automatically  reset  to  the  most  recent  addition.  If  a  user  were  scrolling  back  through  a  list  of  entries  to  monitor 
previous  activity,  he  would  suddenly  find  himself  looking  at  the  end  of  the  list  after  a  refresh.  Also,  system  response 
time  was  so  slow  that,  when  the  system  refreshed,  the  visible  screen  would  be  blank  for  a  period  of  time. 

CommandNet  provided  ways  to  alleviate  these  problems.  The  refresh  rate  could  be  set  through  the  user  Preferences. 
Alternatively,  a  more  immediate  method  was  provided  by  a  Disable  Refresh  button  located  in  the  menu  at  the  top  of 
each  screen.  When  a  user  clicked  on  the  Disable  button,  refresh  was  disabled  until  the  user  clicked  on  the  Enable 
Refresh  button.  The  status  of  disabled  refresh  was  indicated  by  red  text  across  the  top  of  the  list.  While  refresh  was 
disabled,  no  updates  would  appear. 


IStrong  Angel 

In  Strong  Angel  logs,  the  use  of  the  enabling/disabling  refresh  and  changing  refresh  rate  feature  was  minimal. 
Refresh  was  disabled  only  5  times  (via  the  Disable  Refresh  button),  and  refresh  was  enabled  a  total  of  1 1  times  (via 
the  Enable  Refresh  button).  The  refresh  rate  was  changed  5  times  (2  of  those  times  by  a  CMI  person).  76%  of  the 
refresh  activity  occurred  in  the  CMOC  Log,  24%  in  CMOC  Afloat,  and  none  in  the  CA  Log.  It  is  curious  to  note 
that  the  Enable  Refresh  button  was  ued  more  than  the  Disable  Refresh  button.  Since  these  two  buttons  essentially 
act  as  toggles  between  states,  it  appears  that  users  may  have  thought  that  the  Enable  Refresh  button  acted  as  a  forced 
refresh. 


IRIMPAC2000 

The  Enable  Refresh  button  was  used  much  more  often  than  the  Disable  Refresh  button.  Again,  users  must  have 
thought  that  the  Enable  Refresh  acted  as  a  forced  refresh.  Most  of  the  refresh  activity  occurred  between  the  during 
the  war  exercises;  92%  of  the  refresh  enabling  and  82%  of  refresh  rate  changing  occurred  during  that  period. 

Activity  over  other  days  was  minimal. 

One  user,  in  particular,  made  74%  of  all  the  Enable  Refresh  actions.  This  user  was  also  responsible  for  35%  of  all 
refresh  rate  changes.  The  user  was  a  monitor  the  Intel  log  (he  did  not  make  any  entries).  There  were  periods  of  time 
when  the  user  clicked  on  the  Enable  Refresh  button  every  couple  of  minutes.  Since  he  never  disabled  the  refresh,  he 
must  have  assumed  that  the  Enable  Refresh  button  would  activate  immediate  screen  refresh.  This  user’s  activity 
contributed  to  the  Enable  Refresh  peak  in  the  Intel  log  in  the  figure  below. 


About  one  fourth  of  all  RIMPAC 
2000  users  changed  the  refresh 
rate  through  the  user  preferences. 
However,  only  9%  of  the  user 
population  changed  the  rate  more 
than  once.  Yet,  each  time  a  user 
logged  in,  the  refresh  rate  was 
reset  to  the  default  of  60  seconds. 

The  most  popular  setting  for  the 
refresh  rate  were  60  seconds 
(22%),  180  seconds  (21%),  and 
120  seconds  (13%). 


Figure  36  Comparing  use  of  refresh  features  across  logs 
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Preferences 


iStronQ  Anflel 

In  the  Strong  Angel  logs,  once  the  preferences  screen  was  opened,  there  was  a  64%  chance  the  preferences  would  be 
changed. 


Frequency  of  Viewing  /  Changinc 


□  view  prefs  □  change  prefs 


Figure  37  Frequency  of  Viewing  and  Changing  Preferences  in  Strong  Angel  Logs 


Log 

#  times  preferences 
were  viewed 

#  times  preferences 
were  changed 

Likelihood  that 
preferences  would  be 
changed  after  viewing 

CMOC 

10 

5 

50% 

CMOC  Afloat 

4 

4 

100% 

Civil  Affairs 

0 

0 

Overall 

14 

9 

64% 

Table  11  Likelihood  that  preferences  would  be  changed  after  viewing  in  Strong  Angel  Logs 


Log 

%  Users  who  viewed 
preferences 

%  Users  who  changed 
prefererences 

CMOC 

20% 

15% 

CMOC  Afloat 

5% 

5% 

Civil  Affairs 

0% 

0% 

Overall 

25% 

20% 

Table  12  Percentage  of  users  viewing  and  changing  preferences  in  Strong  Angel  logs 
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Use  of  Preferences  by  Log 


□  view  prefs  □  change  prefs 


□  view  prefs  □change  prefs 


Figure  38  Preference  Events  by  Log 


Figure  39  Users’  Preference  Activity  by  Log 


Preference  Settings 

CommandNet  provided  the  option  of  changing  the  order  entries  were  displayed.  By  default,  entries  were  displayed 
most  recent  entry  first,  least  recent  entry  last.  No  users  changed  the  entry  display  order. 

Users  of  Strong  Angel  logs  never  changed  whether  or  not  importance,  date,  time,  or  category  were  visible.  The  only 
person  to  change  the  number  of  days  displayed  was  a  CMI  user.  He  changed  it  from  the  default  of  2  days  to  show 
all  the  entries. 


CommandNet  also  allowed  the  user  to  select  how  the  time  was  displayed.  The  user  could  select  various  time  offsets 
such  as  Zulu,  Kilo,  and  Whiskey.  By  default,  the  offset  was  Zulu.  The  time  offset  was  changed  10  times  by  5 
different  users.  70%  of  these  changes  were  to  Whiskey  time,  and  20%  to  Zulu  time.  There  was  just  one  change  was 
to  Tango  time  (Tucson  or  San  Diego),  but  that  was  probably  a  user  error  since  the  time  offset  was  changed  to 
Whiskey  time  one  minute  later. 


IRIMPAC  2000 

The  pattern  of  viewing  the  preferences  screen  and  changing  preferences  follows  the  pattern  of  usage  during 
RIMPAC  exercises.  (See  Figure  40,  below.) 
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Figure  40  Frequency  of  Viewing  /  Changing  Preferences  in  RIMPAC  2000  Logs 

Once  the  preferences  screen  was  opened,  there  was  about  a  70%  chance  that  the  user  would  make  a  change  to  the 
preferences. 


About  half  the  users  (55%)  changed  preferences  at  least  once.  Half  of  those  users  were  using  the  BWC  log,  and  just 
over  a  quarter  were  using  the  Intel  log.  (See  Table  13,  below.) 
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Log 

%  of  all  users 
who  viewed 
preferences 

%  of  all  users 
who  changed 
preferences 

BWC 

49% 

22% 

JCCC 

8% 

4% 

Intel 

27% 

13% 

CFACC 

14% 

5% 

CECG 

2% 

1% 

Overall 

55% 

46% 

Table  13  Percentage  of  users  viewing  and  changing  preferences  in  RiMPAC  2000  iogs 

Most  of  the  changes  in  preferences  occurred  in  the  BWC  log  and  the  Intel  log,  but  users  of  the  other  logs  were  more 
likely  to  make  a  change  once  they  opened  the  preferences  screen. _ _ 


Log 

#  times  preferences 
were  viewed 

#  times  preferences 
were  changed 

Likelihood  that 
preferences  would  be 
changed  after  viewing 

BWC 

99 

63 

64% 

JCCC 

16 

15 

94% 

Intel 

108 

73 

68% 

CFACC 

17 

15 

88% 

CECG 

7 

6 

86% 

Overall 

247 

183 

70% 

Tabie  14  Likeiihood  that  preferences  would  be  changed  after  viewing  in  RiMPAC  2000  logs 


Figure  41  Preference  Events  by  RiMPAC  2000  Log  Figure  42  Users’  Preference  Activity  by  Log 


Preference  Settings 

The  preference  feature  allowed  users  to  change  the  order  the  entries  were  displayed.  By  default,  entries  were 
displayed  most  recent  entry  first,  least  recent  entry  last.  No  users  changed  the  entry  display  order. 

The  user  could  also  select  how  the  time  was  displayed.  Available  time  offsets  included  Zulu,  Kilo,  and  Whiskey. 

By  default,  the  offset  was  Zulu.  Users  changed  the  offset  to  Whiskey  (Hawaii  time)  81%  of  the  time,  Zulu  13%,  and 
Kilo  6%. 

The  time  offset  was  changed  a  total  of  32  times  by  17  users.  Most  of  those  users  set  the  time  offset  only  once  or 
twice,  except  for  U238,  who  set  it  6  times  in  the  CFACC  log.  This  particular  user  did  the  following: 

•  changed  time  offset  from  Zulu  to  Whiskey 

•  changed  time  offset  from  Whiskey  to  Kilo  5  days  later 

•  reset  the  offset  from  Kilo  to  Whiskey  about  an  hour  later 

•  changed  time  offset  from  Whiskey  to  Zulu  the  next  day 

•  reset  the  offset  from  Zulu  to  Whiskey  immediately  afterwards 

•  reset  the  offset  from  Whiskey  to  Zulu  2  hours  later 
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Help 

We  have  no  data  on  the  help  feature.  Help  pages  were  added  after  the  automated  data  logging  was  implemented. 
No  data  was  collected  on  help. 


Hyperlinks 


Strong  Angel 

In  the  Strong  Angel  logs,  there  were  six  hyperlinks  added  to  entries.  4  of  those  links  were  tests.  The  other  two  were 
the  same  link  (duplicated  across  two  logs)  to  a  news  report. 

RIMPAC  2000 

In  the  RIMPAC  logs,  the  hyperlink  feature  was  used  just  three  times,  all  occurring  before  the  war  exercises  began. 
One  user  was  probably  exploring  the  feature  when  he  made  a  test  entry  and  added  a  URL.  The  second  instance  was 
another  test  entry  by  a  CMI  user.  This  may  have  been  done  for  training  or  demonstrative  purposes.  The  third  use  of 
the  hyperlink  feature  was  the  only  “real”  use.  A  link  was  added  to  an  FBIS  (Federal  Bureau  of  Information 
Services)  report. 


Miscellaneous  Observations 


Log  Content  and  Style 

Use  of  the  logs  differed  between  the  civil-military  Strong  Angel  HA/DR  exercise  and  the  exclusively  military 
RIMPAC  2000  exercises.  Use  also  differed  among  user  communities  within  each  exercise.  The  differences 
included: 

•  usage  consistency 

•  content 

•  style  (all  capitals  versus  lower  case  and  milspeak  versus  natural  English) 

Usage  consistency  was  addressed  partly  in  the  subsection  on  Entries  (frequency  of  entries)  and  addressed  again, 
below,  in  the  subsection  on  shift  changes.  We  recap  these  findings  here. 

Content  was  partly  addressed  in  the  Categories  subsection. 

Here  we  summarize  the  differences  between  exercise  groups  and  among  user  groups  within  each  exercise  group. 

Strong  Angel 

Usage  Consistency 

CommandNet  was  in  use  for  not  more  than  several  days  during  Strong  Angel.  The  CMOC  Afloat  log  and  the  Civil 
Affairs  log  were  not  available  for  more  than  a  day  or  two,  and  the  Civil  Affairs  log  suffered  from  intermittent 
connections. 

Content 

Strong  Angel  participants  used  CommandNet  to  record  mostly  situation  reports  (47%)  and  tracking  events,  both 
people  and  resources  (35%).  There  was  also  significant  use  for  troubleshooting,  chat,  request  for  action, 
acknowledgment,  and  testing  (all  >  10%). 

Style 

All  Strong  Angel  logs  were  written  in  natural  English,  in  lower  case  letters. 

RIMPAC  2000 

Usage  Consistency 

The  BWC,  JCCC,  and  Intel  logs  were  used  throughout  the  entire  month.  The  CFACC  and  CECG  logs  were  used 
only  during  the  war  exercises. 
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The  JCCC  log  was  used  most  consistently;  about  35  entries  were  made  per  day  at  about  16  words  per  entry.  Shift 
changes  were  logged  fairly  consistently. 

Both  the  BWC  and  Intel  logs  had  marked  increase  during  the  war  exercises.  Activity  in  the  BWC  log  increased 
from  about  20  entries  a  day  to  about  60  a  day,  with  no  significant  change  in  the  average  number  of  words  per  entry. 
Activity  in  the  Intel  log  changed  drastically  from  fewer  than  10  entries  a  day  to  a  range  between  65  and  125  a  day. 
There  was  no  significant  change  in  the  average  number  or  words  per  entry.  Shift  changes  in  the  BWC  log  were 
mostly  consistent  at  three  a  day.  There  were  more  shift  changes  logged  in  Intel  during  increased  activity. 

Usage  of  the  CFACC  log  during  its  short  period  of  existence  was  fairly  consistent.  No  shift  changes  were  logged. 

Activity  consistency  in  the  CECG  log  was  similar  to  that  of  the  CFACC  log  but  for  one  exception;  exactly  two  shift 
changes  were  logged  per  day. 

Overall,  the  Intel  log  was  used  the  heaviest  for  making  entries,  and  the  BWC  entries  were  the  wordiest. 

Content 

The  BWC  log  contained  admiral  activity  and  was  an  operational  log.  The  log  can  be  grouped  into  three  content 
periods:  before,  during,  and  after  the  exercises.  Before  the  exercises,  entries  were  mainly  about  trouble  shooting, 
personnel,  tasking,  and  scheduling.  During  the  exercises,  entries  shifted  towards  sitreps,  tracking  of  forces,  reports, 
and  casualties.  A  portion  of  these  entries  contained  information  from  other  logs  (CFACC,  Intel,  and  CECG)  and 
also  notification  that  information  had  been  forwarded  to  other  logs.  The  after-exercises  entries  were  mostly  situation 
reports. 

The  Intel  log  can  also  be  grouped  into  the  same  three  content  periods.  Before  the  exercises,  entries  were  reports 
(often  introduced  by  “Sources  indicate  that. . .”),  general  information,  and  information  on  the  enemy  forces.  During 
the  exercises,  entries  were  sitreps  and  tracking  information.  Some  of  the  entries  were  Intel  messages  pasted  into 
CommandNet.  (This  was  apparent  by  the  style  and  capitalization  of  the  messages.)  The  after-exercises  entries  were 
mostly  tracking  information  from  messages. 

The  JCCC  log  included  maintenance  entries,  trouble  shooting,  and  system  and  network  reports. 

CFACC  entries  during  the  exercises  logged  exercise  events  and  tactical  details:  sitreps,  aircraft  tracking,  enemy 
reports,  and  battle  reports.  At  the  end  of  the  exercises,  entries  were  sitreps,  tasking,  and  about  enemy  forces. 

CECG  entries  during  the  exercises  were  specific  to  specific  exercise  events,  damage  reports,  munitions,  and 
targeting.  Entries  also  contained  information  on  the  enemy,  warnings  and  advisories,  and  commands.  Towards  the 
end  of  the  exercises,  entry  contents  dealt  more  with  tracking  of  both  friendly  and  enemy  forces. 

Style 

Almost  all  log  entries  were  in  lower  case  with  capitalized  abbreviations  denoting  operations,  teams,  and  logs. 
Language  was  mostly  natural.  The  JCCC  log  used  some  “tech  speak.” 

The  one  exception  was  Intel  messages  pasted  into  the  Intel  log  in  capital  letters.  Reports  and  manually-typed 
information  were  in  lower  case  and  natural  language. 

There  were  considerable  typographic  errors  in  the  logs.  A  spell-checking  tool  was  requested  on  numerous 
occasions. 


Shifts 


Strong  Angel 

There  were  no  shift  changes  logged  in  the  Strong  Angel  logs. 

RIMPAC  2000 

In  the  RIMPAC  2000  logs,  entries  reporting  shift  changes  were  not  made  consistently.  Sometimes  shift  changes 
were  reported  by  the  user  leaving  the  watch,  and  some  were  reported  by  the  user  assuming  the  watch.  There  was 
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often  some  overlap,  and  both  events  were  recorded.  62  “stand  down”  entries  were  reported  and  164  “stand  up” 
entries. 


Frequency  of  Shift  Changes 
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Figure  43  Frequency  of  Shift  Changes  in  RiMPAC  2000  Logs 


For  purposes  of  keeping  a  searchable  permanent  record,  the  category  feature  would  have  leant  itself  well  to  labeling 
shift  changes.  However,  categories  were  never  used  for  shift  changes.  In  addition,  since  the  language  varied  in  each 
entry  (“x  assumed  the  position  of’,  “x  relieved”,  “y  was  relieved  by  x”,  etc),  there  was  no  easy  way  to  search  and 
locate  entries  with  shift  changes.  Why  were  categories  not  used  to  label  shift  changes?  Perhaps,  since  no  category 
was  initially  created  for  shift  changes,  users  had  nothing  appropriate  to  choose. 


“Routine”  importance  was  used  for  shift  changes  in  all  cases  except  one  use  of  “Immediate”  for  an  unexceptional 
entry. 


All  BWC  shift  entries  were  “stand  ups”  (made  by  the  person  assuming  the  watch).  All  shift  entries  were  made  by 
bwc  (battle  watch  officer)  or  abwc  (assistant  battle  watch  officer).  There  were,  on  average,  3  shift  entries  per  day. 
BWC  shifts  were  every  6  hours,  or  4  per  day,  so  shift  changes  were  not  logged  consistently. 


Figure  44  shows  the  frequency  of  logged  shift  changes,  broken  out  by  log. 


Frequency  of  Shift  Changes  by  Log 


Day 


All  CECG  shift 
entries  were  “stand 
ups.”  All  shift 
entries  were  made  by 
cecgwo  (CECG 
watch  officer). 

There  were  exactly  2 
shifts  per  day  in  this 
log.  This  coincides 
with  the  12-hour 
shifts. 

The  JCCC  log 
contained  the  highest 
frequency  of  shift 
changes,  about  5  a 
day. 


Figure  44  Frequency  of  Shift  Changes  by  Log  in  RIMPAC  2000 
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The  Intel  log  averaged  3  shift  change  entries  per  day.  The  frequency  of  shift  changes  appears  to  be  consistent  with 
the  frequency  of  entries  in  this  log.  In  other  words,  as  activity  increased,  so  did  the  number  of  shift  changes. 

Shift  changes  were  not  entered  in  the  CFACC  log.  There  should  have  been  2  entries  per  day  for  the  12-hour  shifts. 

There  were  about  12  shift  change  entries  per  day  except  for  on  days  6,  8,  10,  11,  14,  and  19.  (Day  19  had  only  three 
entries  total.) 


Replicating  Entries  Across  Logs 

We  found  several  instances  of  replicated  entries  in  multiple  logs.  There  are  two  different  cases  where  this  happened. 
In  one  instance,  a  user  decided  to  submit  the  same  entry  to  more  than  one  log.  In  the  second  situation,  someone 
monitoring  a  particular  log  wanted  to  pass  on  a  specific  entry  to  those  monitoring  another  log. 

In  order  to  replicate  an  entry  and  post  it  to  multiple  logs,  a  user  would  either  have  to  copy  and  paste  the  text  or  type 
it  in  more  than  once.  A  potential  new  feature  of  CommandNet  would  facilitate  cross  posting  to  multiple  logs. 
Perhaps  the  display  could  reflect  from  which  log  the  replicated  entry  was  initially  posted.  Just  as  useful,  something 
in  the  original  log  could  indicate  that  the  entry  had  been  copied,  for  reasons  of  security. 

Strong  Angel 

After  the  CMOC  Afloat  log  was  created  on  June  12*,  12  entries,  dated  June  1 1*,  were  copied  and  pasted  from  the 
CMOC  log.  A  built-in  feature  would  have  made  the  task  considerably  easier.  In  addition,  it  might  have  been  useful 
for  the  users  of  the  CMOC  Afloat  log  to  know  the  origin  and  date  of  the  copied  entries. 

RIMPAC  2000 

The  RIMPAC  2000  logs  have  numerous  examples  of  entries  proceeded  by  “Got  info  from  Intel”  or  appended  by 
“BWC  notified”  or  “Tech  Control  informed.”  Again,  these  would  be  good  arguments  for  a  cross  posting  capability 
that  indicated  where  an  entry  was  copied  to  and  also  where  the  copied  entry  had  been  taken  from. 


Profiles  of  User  Groups 


Strong  Angel 


CMOC 

CMOC  Afloat 

Civil  Affairs 

#  Days  in  use 

6 

3 

1 

%  of  entries  made 

62% 

18% 

20% 

#  Users 

12 

2 

4 

#  Users  making  entries 

12 

2 

4 

#  Main  users 

4  (64%  entries) 

2  (100%  entries) 

3  (96%  entries) 

Average  #  entries  per  day 

12 

7 

22 

Average  #  entries  per  day  per  user 

4 

8 

7 

%  duplicate  entries 

0 

0 

0 

Training 

yes 

Likes 

ease  of  use 

Requests 

chat,  ability  to  include  attachments 

Complaints 

refresh,  categories  not  used  properly 

used  categories 

yes  (53%) 

no 

yes  (91%) 

used  importance 

yes 

very  little 

no 

looked  at  headlines 

yes,  a  little 

no 

yes 

looked  at  searches 

yes,  a  little 

no 

yes 

changed  preferences 

yes 

yes 

yes 

enabled  or  disabled  refresh 

yes 

yes 

yes 

Table  15  Overview  of  Strong  Angel  logs,  their  usage,  and  user  feedback 

Strong  Angel  participants  that  responded  to  our  survey  all  had  experience  using  computers.  All  were  familiar  with 
some  type  of  collaborative  software,  ranging  from  email  and  shared  directories  to  audio  and  video.  Half  the 
participants  had  used  GroupSystems  before. 
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Other  methods  of  recording  observations  and  accessing  information  included  local  logs  (greenbook),  email,  and  the 
“collaborative  logbook”,  an  older  version  of  CommandNet. 

Strong  Angel  participants  differed  from  participants  in  RIMPAC  2000  exercises  in  that  Strong  Angel  users  received 
more  training  from  CMI  staff.  Overall,  they  were  pleased  with  the  training  they  received  and  found  the  tool  easy  to 
use. 


'‘Very  straightforward,  easy  to  understand.  ” 

“It  was  good.  They  showed  me  the  basics,  and  if  I  had  any  questions,  I  knew  who  was 
around  to  ask.  1  mean  if  I  was  in  my  office  by  myself,  1  could  probably  figure  it  out  on  my 
own.  I  think  it's  very  simple  so  you  don't  need  a  lot  of  training  which  is  what  I  like  about  it.  ” 

“We  are  learning  as  we  go.  ” 

Strong  Angel  participants  used  CommandNet  to  record  mostly  situation  reports  (47%)  and  tracking  events,  both 
people  and  resources  (35%).  There  was  also  significant  use  for  troubleshooting,  chat,  request  for  action, 
acknowledgment,  and  testing  (all  >10%). 

The  category  feature  was  used  consistently  (91%)  in  the  Civil  Affairs  log,  about  half  the  time  in  the  CMOC  log,  and 
not  at  all  in  the  CMOC  Afloat  log.  Surveyed  participants  had  mixed  reactions  to  the  use  of  categories. 

“Could  be  useful  if  we  used  it.  Problem  is  we  don't  use  it  well.  “ 

Someone  else  stated,  “not  much  use  for  it.  ” 

On  the  use  of  the  importance  features,  users  noted  that  there  was  little  need  to  prioritize  entries  higher  than 
‘Routine.’ 


“1  haven't  had  to  change  it;  (everything  is  routine  so  far).  ” 

“I  just  kept  it  as  routine,  did  not  really  notice  it.  ” 

The  search  feature  was  little  used.  Some  of  the  participants  thought  it  was  a  good  idea  to  have,  but  either  had  no 
reason  to  use  it  or  felt  there  was  not  enough  material  to  warrant  a  search. 

“Reusability  of  saved  searches  is  an  important  feature.  ” 

“I  heard  from  x  there  is  a  search.  Thought  that  was  a  good  idea,  but  never  used  it  yet.  ” 

The  headline  feature  was  not  used  much  either.  For  some  it  was  a  matter  of  training,  others  were  not  even  aware  the 
feature  existed. 

Overall,  Strong  Angel  participants  surveyed  said  that  CommandNet  was  easier  to  use  than  other  comparable 
systems.  They  found  it  useful,  appropriate  for  the  exercise,  and  easy  to  use. 

When  asked  what  they  found  easiest  about  CommandNet,  one  user  said  “ease  of  getting  a  large  group  of  people  to 
log  on  without  a  lot  of  training  and  get  usefulness  out  of  it.''  Another  user  found  it  difficult  to  find  information  in 
the  logs. 


“I  had  the  most  trouble  with  the  ability  to  go  back  and  review  things.  Reviewing  multiple 
days  is  hard  to  make  sense.  Operators  are  trained  on  key  words  but  it  depends  on  what  you 
are  logged  into.  There  is  no  degree  of  assurance  that  you  would  find  what  you  are  looking 
for.  ” 

Most  commonly  requested  features  included  chat  or  a  log  reserved  for  side  conversations,  and  the  ability  to  add 
attachments. 

Below  are  two  graphs  representing  typical  usage  of  a  single  participant  during  Strong  Angel.  Activiy  was  minimal 
as  the  logs  were  in  use  for  not  more  than  six  days. 
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General  Activity 


_ Day _ 

Figure  45  General  activity  of  typical  user 


Frequency  of  Entries 


Day 

Figure  46  Frequency  of  Entries  for  typical  user 


RIMPAC  2000  I 

BWC 

JCCC 

Intel 

CFACC 

CECG 

#  Days  in  use 

21 

19 

11 

7 

8 

%  of  entries  made 

24% 

26% 

29% 

12% 

8% 

#  Users 

68 

19 

28 

18 

5 

#  Users  making  entries 

8 

14 

13 

6 

1 

#  Main  users 

2  (94  entries) 

7  (96%  entries) 

6  (97%  entries) 

4  (99%) 

1  (100% 
entries) 

Average  #  entries  per  day 

29 

35 

67 

44 

24 

Average  #  entries  per  day  per 
user 

25 

7 

16 

14 

24 

%  duplicate  entries 

3% 

3% 

2% 

3% 

2% 

typical  user 

data  entry, 
monitor 

data  entry 

43%  monitor, 

57%  data  entry, 
used  BWC 

data  entry 

data  entry 

training 

on  the  job 

on  the  job 

on  the  job 

on  the  job 

on  the  job 

log  review 

yes 

at  beginning  of 
watch, 

sometimes  at 
end 

yes 

yes 

2-3  times  per 
watch 

likes 

scrollback 

feature, 

persistent 

history 

timestamps,  #’d 
entries 

timestamps, 
importance,  user 
id  associated  wit 
entry,  history 

timestamps, 
viewing  other 
entries,  viewing 
other  logs 

timestamps, 

#’d  entries, 
viewing  other 
logs 

requests 

chat,  ability  to 
edit  or  delete 
entries 

ability  to  edit 

chat,  multiple 
logs  open 
simultaneously 

ability  to  edit 
entries 

complaints 

slow 

slow,  not  enough 
training 

no  one  looked  at 
logs 

slow,  refresh 
problem 

used  categories 

2% 

no 

77% 

no 

no 

used  importance 

no 

no 

yes 

yes 

no 

looked  at  headlines 

no 

no 

50%  users 
looked,  6%  did 
something 

no 

no 

looked  at  searches 

no 

no 

44  %  looked, 

31%  did 
something 

no 

no 

changed  preferences 

yes 

no 

yes  (44%  users) 

no 

yes 

enabled  or  disabled  refresh 

yes 

no 

yes  (31%) 

no 

no 

Table  16  Overview  of  RIMPAC  2000  logs,  their  usage,  and  user  feedback 
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BWC 

The  Battle  Watch  Captain’s  log  was  used  as  an  operational  log.  The  log  could  be  grouped  into  three  content 
periods:  before,  during,  and  after  the  exercises.  Before  the  exercises,  entries  were  mainly  about  trouble  shooting, 
personnel,  tasking,  and  scheduling.  During  the  exercises,  entries  shifted  towards  sitreps,  tracking  of  forces,  reports, 
and  casualties.  A  portion  of  these  entries  contained  information  from  other  logs  (CFACC,  Intel,  and  CECG)  and 
also  notification  that  information  had  been  forwarded  to  other  logs.  The  after-exercises  entries  were  mostly  situation 
reports. 

The  BWC  log  was  used  throughout  the  entire  month,  during  both  Strong  Angel  and  RIMPAC  exercises.  There  were 
typically  about  22  entries  a  day,  but  activity  increased  from  about  20  entries  a  day  to  60  during  the  exercises.  50% 
of  all  entries  were  made  by  bwc,  and  44%  were  made  by  abwc.  These  roles  were  used  by  potentially  16  different 
people.  The  graphs  below  show  activity  and  entry  patterns  for  both  bwc  and  abwc  users.  Note  the  shift  in  entry 
activity  from  the  bwc  to  abwc  during  RIMPAC. 

(Note  that  graphs  in  the  Log  Profile  subsection  use  the  same  scales  for  visual  comparison.  In  other  words,  all 
“Overall  Activity”  graphs  are  scaled  to  2000  events  per  day,  and  all  “Frequency  of  Entries”  graphs  are  scaled  to  50 
events  per  day.) 


Overall  Activity:  bwc 


Day 


Figure  47  Overall  activity  of  the  ‘bwc’  user 


Figure  48  Frequency  of  entries  made  by  ‘bwc’ 


All  BWC  users  surveyed  were  regular  computer  users.  All  had  used  email  and  shared  directories,  80%  had  used 
chat,  and  60%  were  familiar  with  GroupSystems.  Before  using  CommandNet  as  a  means  of  reporting  observations 
and  accessing  information,  participants  had  used  a  local  log  (the  greenbook),  email,  and  a  previous  version  of 
CommandNet  referred  to  as  “the  collaborative  logbook.” 

They  thought  the  category  feature  was  useful  enough  and  easy  to  use,  but  it  was  used  an  insignificant  number  of 
times.  The  importance  feature  was  used  minimally  as  well.  No  one  surveyed  used  search  or  headlines. 

Training  was  rated  as  neither  adequate  nor  inadequate.  Most  everyone  learned  basic  features  from  watching 
someone  else.  One  participant  complained  that  he  had  received  some  training  -  well  after  he  had  begun  to  use  the 
tool. 
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All  thought  CommandNet  was  relatively  easier  to  use  than  past  methods  (2  on  a  scale  of  5,  with  1  being  the  easiest). 
Most  rated  CommandNet  as  somewhat  slower  than  other  systems. 

Overall,  BWC  users  surveyed  judged  CommandNet  to  be  somewhat  useful,  somewhat  easy  to  use,  and  very 
appropriate  to  their  job. 

What  users  liked  most  about  CommandNet  were  ease  of  use  and  its  value  as  a  permanent  record  of  events.  They 
commented  on  how  easy  it  was  to  add  entries  and  how  easy  and  useful  the  scroll  back  feature  was. 

Users  also  wanted  the  ability  to  delete  or  edit  entries. 


JCCC 

The  JCCC  log  was  created  for  use  in  the  Joint  Command  Control  Center.  The  JCCC  log  was  used  for  technical 
support:  logging  troubleshooting  events,  maintenance,  and  system  and  network  status.  The  users  in  the  JCCC 
community  were  all  regulars  and  were  accustomed  to  logging  entries  in  other  tools  like  an  Excel  spreadsheet.  Word 
document,  or  local  logs  (greenbook).  All  had  used  email,  and  75%  had  used  chat,  shared  directories,  and  audio. 
None  of  the  users  interviewed  had  received  any  training;  they  learned  to  login  and  make  entries  from  someone  on 
the  previous  watch. 

The  log  was  in  use  for  16  days.  Figure  51  displays  the  overall  activity  for  a  typical  user  in  the  JCCC  Log.  Overall 
activity  includes  entries  made,  use  of  the  advanced  features,  screen  refreshes,  navigational  activity,  etc.  The  activity 
level  is  fairly  constant  throughout  its  use.  Figure  51  shows  the  frequency  of  entries  logged  for  a  typical  user.  A 
typical  user  also  averaged  about  seven  entries  a  day. 
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Figure  51  Overall  activity  for  a  typical  user 


Figure  52  Typical  frequency  of  entries 


Compared  to  methods  utilized  in  the  past,  JCCC  users  found  CommandNet  ''easy  to  use''  and,  according  to  one 
person,  "about  the  best  I’ve  seen."  They  liked  the  fact  that  the  entries  were  numbered  because  they  could  refer  to 
them  by  the  number.  Indeed,  about  3%  of  the  entries  contained  references  to  other  entries.  Interviewed  users  also 
liked  the  automatic  time  stamps  associated  with  each  entry. 


Another  well-liked  feature  was  the  ability  to  scroll  back  through  the  history  of  logged  events.  Users  reportedly 
reviewed  the  log  at  the  beginning  of  their  shift  and  sometimes  also  at  the  end. 


Not  one  user  complained,  during  our  interviews,  about  the  slowness  of  the  system.  That  was  surprising  considering 
that  5%  of  the  entries  were  duplicates  resulting  from  hitting  the  submit  button  more  than  once  before  the  system  had 
time  to  update  the  logs.  The  JCCC  log  had  a  higher  percentage  of  duplicate  entries  than  any  other  log. 


More  than  one  user  requested  the  ability  to  edit  entries;  there  were  lots  of  typographical  errors.  One  user  did  not 
understand  the  symbols  that  represented  the  level  of  importance  (i.e.,  R,  P,  O,  and  Z.  See  the  Importance 
subsection).  That  user  would  like  to  see  help  on  the  symbols  or  a  visible  legend. 

Other  than  for  simple  data  entry,  the  JCCC  user  community  did  not  use  CommandNet  for  anything  else.  The  easy- 
to-use  category  and  importance  features  were  not  used  to  annotate  entries,  and  none  of  the  more  advanced  features 
(search,  headlines)  were  used  at  all.  A  few  users  viewed  and  changed  the  user  preference  settings,  but  the  refresh 
feature  was  not  used. 
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They  found  it  somewhat  easier  than  other  tools.  Overall,  it  was  rated  as  somewhat  useful,  somewhat  appropriate, 
and  somewhat  easy  to  use. 


Intel 

The  Intel  log  contained  more  entries  than  any  other  log.  It  was  used  heavily  during  the  RIMPAC  exercises.  In  fact, 
activity  increased  drastically  from  fewer  than  10  entries  a  day  to  a  range  between  65  and  125  a  day.  The  graphs 
below  show  typical  patterns  of  a  single  Intel  log  user.  In  addition  to  making  entries  in  the  Intel  log,  the  typical  Intel 
user  monitored  the  BWC  log. 


Overall  Activity 
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It 


Figure  53  Overall  activity  of  an  Intel  user 


Figure  54  Frequency  of  entries  made  by  ‘jiseswo’ 


Before  the  RIMPAC  exercises,  entries  were  reports  (often  introduced  by  “Sources  indicate  that...”),  general 
information,  and  information  on  the  enemy  forces.  During  the  exercises,  entries  were  sitreps  and  tracking 
information.  Some  of  the  entries  were  clearly  Intel  messages  pasted  into  CommandNet.  (This  was  apparent  by  the 
style  and  capitalization  of  the  messages.)  The  after-exercise  entries  were  mostly  tracking  information  copied  from 
messages. 


At  least  one  third  of  the  surveyed  population  was  reservist.  All  were  computer  users  and  familiar  with  email  and 
shared  directories.  Other  comparable  methods  for  making  observations  and  accessing  information  were  Microsoft 
Excel  spreadsheet,  Microsoft  Word,  local  logs  (greenbook),  and  older  versions  of  CommandNet. 


No  one  received  training;  two  users  said  they  figured  things  out  on  their  own,  but  one  complained  that  he  needed 
more. 


Unlike  other  logs,  the  category  feature  was  used  heavily,  and  most  (67%)  found  it  both  useful  and  easy  to  use.  Out 
of  nine  categories  created,  seven  were  used  frequently.  The  importance  feature  was  used  regularly  in  the  Intel  logs, 
too,  but  was  rated  as  less  than  useful. 

“7  don't  prioritize  reading  based  on  that  since  I  read  them  as  they  come  in.  I  don't  do  this 
often  enough  to  remember  what  ‘O’  and  ‘P’  mean.  ” 

A  couple  of  users  made  several  searches  more  than  once.  Searches  were  rated  useful,  helpful,  and  easy  but  were 
considered  slow  and  not  so  intuitive.  One  person  suggested  that  it  might  be  easier  to  use  the  ‘Find’  feature  of  the 
browser. 


Headlines  were  not  used,  but  a  few  people  looked  at  them. 

Most  users  said  CommandNet  was  somewhat  easier  than  other  systems  and  not  any  faster.  Everyone  interviewed 
thought  it  provided  better  analysis.  Overall,  they  unanimously  found  it  useful,  appropriate,  and  easy. 

Below  are  some  comments  about  what  the  Intel  community  liked. 

“auto  log  entry  time,  precedence  level,  originator,  as  much  space  as  needed  to  write 
whatever” 
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‘'The  fact  that  it's  quick,  easy  to  use,  research  tool  for  current  events.  It's  a  log.  Recap 
events  go  back  and  figure  out  what's  going  on.  Keeps  key  nodes  on  ship  apprised  of  what's 
going  on.  ” 

“Very  intuitive  ui.  It  gives  me  access  to  real  time  info.  ” 

Common  complaints  were  about  the  frustrating  refresh  and  slowness  of  the  system. 

Requested  features  included  the  ability  to  force  refresh,  chat,  and  the  capability  of  adding  attachments.  Since 
CommandNet  could  ultimately  be  used  as  a  chat  tool,  we  infer  from  user  comments  that  chat  was  discouraged. 

“I  would  like  more  info.  It  would  be  nice  to  have  a  way  to  ask  questions.  If  I  am  unclear,  I 
send  out  email.  The  log  is  not  meant  to  be  interactive.  ” 


CFACC 

The  Coalition  Forces  Air  Command  Control  Log  was  used  for  just  six  days  during  the  RIMPAC  exercises.  There 
was  little  activity  in  the  log,  about  44  entries  a  day  (14  per  user). 


Two  thirds  of  survey  participants  were  regulars,  one  third  were  reservist.  All  were  users  of  email,  chat,  shared 
directories,  and  document  management  software.  Tools  used  in  the  past  to  record  observations  and  access 
information  included  Microsoft  Word  and  local  logs,  or  the  greenbook. 

Two  thirds  of  the  surveyed  population  received  no  training  on  the  use  of  CommandNet.  The  others  said  that  training 
had  been  adequate. 


Categories  were  not  used.  Two  respondents  said  that  there  were  no  categories  from  which  to  choose. 
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Figure  55  Overall  activity  of  typical  user 


Figure  56  Frequency  of  entries  of  typical  user 


Those  who  used  the  importance  feature  said  it  was  easy  to  use,  but  one  did  not  know  what  the  purpose  was.  One 
user  remarked  that  there  were  no  defined  criteria  for  using  the  importance  feature.  Another  user  indicated  that  the 
importance  feature  was  not  used  because  the  exercise  was  “not  really  a  war.''  About  one  quarter  of  entries  were 
labeled  as  ‘Flash.’  When  asked  if  CommandNet  would  be  useful  as  an  analysis  tool,  one  user  said  it  would  be  if 
categories  and  importance  were  used  properly. 

Search  and  headlines  were  not  used.  One  person  tried  to  use  the  search  feature  but  could  not  get  it  to  “come  up." 
Another  user  thought  that  search  would  be  more  useful  during  a  real  war. 

No  shift  changes  were  recorded  in  CFACC  logs. 

Compared  to  other  systems,  all  CFACC  users  surveyed  found  CommandNet  easier  to  use.  Overall,  CFACC  users 
thought  CommandNet  was  somewhat  useful,  somewhat  appropriate,  and  somewhat  easy. 

Users  liked  the  automatic  timestamp  associated  with  each  entry.  They  liked  being  able  to  see  other  people’s  entries 
and  other  logs  (BWC). 

Most  users  commented  on  the  slowness  of  the  system  and  how  it  would  not  be  used  during  a  war  if  it  were  that  slow. 
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‘'Updates  are  too  slow.  Data  entry  needs  to  be  a  lot  quicker  but  updating  is  too  slow  and  I 
end  up  writing  on  paper.  It's  no  better  than  a  notebook. " 

3%  of  the  entries  were  duplicates  resulting  from  repeated  submitting. 


CECG 

The  CECG  log  was  created  for  the  Coalition  Exercise  Control  Group.  The  purpose  of  the  log  was  for  recording 
significant  exercise  events  and  operational  activities.  Everyone  surveyed  was  a  regular.  All  were  experienced 
computer  users  and  had  used  email,  chat,  and  shared  directories.  Most  had  used  Microsoft  Word  in  the  past  for 
similar  event  logging.  At  least  two  people  had  used  a  precursor  to  CommandNet,  the  “collaborative  log.” 
According  to  our  survey,  no  one  interviewed  received  any  training  on  the  use  of  CommandNet. 

The  log  was  in  use  for  eight  main  days.  All  entries  were  made  by  ‘cecgwo’,  a  role  created  for  the  CECG  watch 
officer.  Based  on  information  extracted  from  logged  shift  changes,  the  cecgwo  role  was  potentially  used  by  five 
different  people.  Eigure  57  shows  the  overall  activity  of  this  role.  Use  is  constant  during  the  eight  days.  The 
frequency  of  entries  made  by  cecgwo,  shown  in  Eigure  57,  follows  a  bell- shaped  pattern;  more  entries  were  made 
during  the  exercises.  About  24  entries  were  made  a  day,  possibly  by  at  least  two  different  people  or  shifts  per  day 
(there  were  two  shift  changes  logged  per  day). 
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Figure  57  Overall  activity  for  user  ‘cecgwo’ 
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Figure  58  Frequency  of  entries  for  user  ‘cecgwo’ 


CommandNet  was  used  by  the  CECG  user  community  mainly  to  log  entries.  People  interviewed  said  they  used  the 
scroll  back  feature  at  the  beginning  of  their  watch  to  review  previous  entries.  There  was  a  belief  expressed  by  two 
users  that  no  one  was  looking  at  the  logs,  although  there  was  some  evidence  that  there  were  people  who  logged  in 
just  to  monitor  the  logs. 

No  features  were  used  other  than  change  in  user  preferences.  The  lack  of  feature  use  could  be  attributed  to  the  lack 
of  training  or  to  the  urgency  of  the  situation. 

“When  it  gets  busy,  I  only  want  to  make  entries.  ” 

Most  popular  positive  comments  about  CommandNet  were  regarding  numbered  entries  and  automatic  timestamps. 
One  person  particularly  liked  the  ability  to  see  other  logs. 

One  user  complained  about  the  refresh  problem  during  review  of  entries.  It  was  unclear  from  the  data  if  the  user 
knew  how  to  disable  the  refresh.  Slowness  of  the  system  was  another  common  complaint.  According  to  one 
participant,  the  slowness  made  CommandNet  unreliable  as  a  means  of  communication:  “when  you  need  it  most, 
you're  unable  to  use  iC 

Two  participants  surveyed  wanted  the  ability  to  edit  entries  and  to  make  corrections.  Someone  else  commented  that 
it  was  good  to  have  a  permanent  record  -  Microsoft  Word  files  could  be  changed. 

Someone  complained  about  the  lack  of  instructions  that  accompanied  the  tool  -  was  the  help  feature  not  available  or 
just  too  unobtrusive?  They  all  said  it  was  somewhat  harder  (4  out  of  a  scale  of  5,  with  5  being  harder)  to  use  than 
other  systems,  and  somewhat  slower. 
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CommandNet  Usability  Report 
Introduction 

This  report  discusses  the  usability  aspects  of  CommandNet,  a  system  developed  by  the  University  of 
Arizona’s  Center  for  Management  of  Information.  The  report  is  based  on  our  observation  of  the  users 
participating  in  Strong  Angel  and  RIMPAC  2000  and  on  their  feedback  as  reflected  in  interviews  and  their 
responses  to  the  survey  questionnaire.  As  we  identify  usability  problems,  we  make  design  suggestions  on 
how  to  fix  them.  First  we  cover  the  usability  issues  essential  to  the  basic  use  of  CommandNet:  speed,  space 
constraints,  menu,  navigation,  preferences,  look  and  feel.  Then  we  address  the  usability  of  the  more 
advanced  features  that  were  used  by  only  a  small  subset  of  users:  search,  headlines,  and  administration. 

Speed 

Although  we  had  positive  feedback  from  the  users  about  CommandNet,  a  major  and  very  serious  area  of 
complaints  was  the  speed  of  the  system.  Sometimes  it  took  the  system  several  minutes  to  refresh  the 
screen.  The  people  making  entries  were  very  concerned  that  they  might  get  a  call  and  have  to  give  a 
situation  update  while  their  screen  was  blank.  We  also  noticed  a  significant  amount  of  duplicate  entries 
resulting  from  impatient  users  clicking  on  the  Submit  button  multiple  times.  Improving  the  speed  of  the 
system  will  make  a  tremendous  difference  on  the  acceptance  of  CommandNet. 


Space  constraints 

During  RIMPAC  2000,  participants  often  used  CommandNet  while  having  several  other  applications 
opened  at  the  same  time.  Due  to  this  constraint,  the  screen  real  estate  available  to  CommandNet  was  not 
sufficient  to  fit  the  entire  user  interface.  Vital  parts  like  the  frame  to  create  new  entries  and  the  menu 
would  often  be  partially  hidden  from  the  user.  Many  users  did  not  use  the  menu  often,  but  a  good  part  of 
their  CommandNet  window  was  occupied  by  it,  and  there  was  no  way  to  control  this  layout.  We  suggest 
the  following  redesign  of  CommandNet’ s  layout. _ 
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Figure  1:  Proposed  CommandNet  Screen  with  menu  exposed 
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Figure  2:  Proposed  CommandNet  screen  with  menu  hidden 

Figure  1  displays  the  screen  the  users  would  see  when  they  chose  a  log.  The  menu  that  is  now  on  top  has 
been  moved  to  the  left.  This  will  help  users  scan  the  menu  options.  Usability  studies  show  that  users  can 
locate  the  menus  better  on  the  left  than  on  top  of  the  window  (Nygren,  1996).  It  is  also  easier  to  scan  the 
menu  options  since  there  is  only  one  option  per  line.  The  “Hide  Menu”  button  allows  users  to  hide  the 
menu,  thus  creating  more  room  for  the  content  (Figure  2). 

In  the  current  interface,  it  is  not  possible  to  see  the  bottom  frame  in  its  entirety  when  the  window  is  shrunk 
to  less  than  half  the  screen  size,  which  happened  quite  often  during  CommandNet’ s  use  during  the 
exercises.  We  suggest  adding  the  “Add  New  Entry”  button  to  the  top  of  the  screen,  which  brings  up  a  pop 
up  window  where  the  user  can  make  a  new  entry.  Since  the  pop  up  window  is  temporary,  it  can  be  made 
an  appropriate  size  to  display  the  entire  entry  form  without  interfering  with  other  windows  on  the  user’s 
screen.  A  pop  up  window  also  allows  the  user  to  refer  to  the  other  entries  while  making  the  new  one. 

This  suggested  new  design  does  not  use  frames.  The  benefits  of  this  will  become  clear  when  we  discuss 
the  navigation. 

Menu 

We  define  two  types  of  links:  navigation  links  and  action  links.  Navigation  links  take  the  user  to  a  different 
page,  while  the  action  links  perform  a  function.  Differentiating  between  these  two  types  of  links 
graphically  helps  the  users  to  find  them  in  the  interface  and  to  infer  the  behavior  of  a  link.  In 
CommandNet’ s  menu,  both  types  of  links  are  present.  Log,  Headlines,  Search,  Preferences,  Admin, 
Logout,  and  Help  are  the  navigational  links  because  they  take  the  user  to  a  different  page.  Refresh 
Enable,  Disable  are  functional  links  because  they  perform  a  function  of  enabling  or  disabling  refresh 
without  going  to  any  other  page.  Making  the  navigation  links  look  different  from  the  action  links  can  help 
the  user  find  the  functionality  they  need  much  more  quickly  and  easily. 

Refresh 

Traditionally,  actions  in  interfaces  are  shown  with  a  button.  Users  are  very  accustomed  to  this  convention 
and  quickly  locate  the  buttons  on  the  screen.  We  suggest  making  a  refresh  button  that  says  “Enable 
Refresh”  if  it  is  disabled,  and  changes  to  “Disable  Refresh”  if  it  is  enabled.  This  is  a  toggle  button.  It 
performs  two  functions  depending  on  the  current  state  of  the  system  (like  a  light  switch).  Such  buttons  are 
usually  very  intuitive  to  use,  and  they  save  space  on  the  screen. 
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It  is  very  important  for  this  button  to  be  visible  to  the  users  because  the  refreshing  of  the  screen  while  users 
were  scrolling  to  review  the  logs  caused  a  lot  of  frustration.  Visible  here  does  not  only  mean  that  this 
button  is  in  the  user’s  field  of  vision,  but  also  that  it  is  easily  noticeable.  To  help  the  user  notice  it,  it  is 
important  to  put  it  in  a  place  where  the  user’s  attention  is  already  focused  -  near  the  log  entries  (Figure  3). 


[Log  Content  goes  here] 


Figure  3:  Proposed  CommandNet  screen  with  refresh  enable/disable  toggle  button 

Navigation 

The  navigation  links  are  now  the  ones  left  in  the  menu.  They  perform  four  important  functions: 

■  Provide  current  orientation 

■  Tell  the  user  how  to  get  back 

■  Expose  the  functionality  to  the  user 

■  Provide  a  way  to  get  to  other  pages 


Provide  current  orientation 

Users  sometimes  felt  disoriented  because  the  menu  options  that  corresponded  to  where  they  were  in  the 
interface  were  not  highlighted  consistently.  After  choosing  a  log,  and  entering  the  main  log  screen 
displaying  the  entries,  users  would  see  none  of  the  menu  options  highlighted.  If  a  user  then  clicked  Search, 
it  was  not  clear  how  to  get  back  to  the  screen  of  entries.  It  was  not  obvious  that  Log  took  one  back  to 
entries  because  it  was  not  highlighted  while  the  user  was  displaying  entries.  If  the  user  then  hit  the  “Back” 
button  on  the  browser,  it  would  take  her  back  to  the  Log  Entries,  but  the  Search  menu  option  would  still  be 
highlighted  because  the  upper  frame  was  not  refreshed. 

There  are  several  design  improvements  to  avoid  such  situations.  A  general  principle  is  to  always  show  the 
menu  option  corresponding  to  the  content  of  the  page  as  highlighted  (even  when  the  user  just  chose  a  log 
and  did  not  click  on  any  menu  options  yet).  If  the  user  hits  the  Back  button,  the  highlighted  menu  option 
should  be  changed  as  appropriate.  Using  the  frames  often  causes  the  problem  of  the  navigation  links  not 
refreshing  when  the  Back  button  is  clicked.  The  new  interface  we  are  proposing  does  not  use  frames, 
making  the  navigation  much  easier  to  implement  correctly.  Clear  terminology  of  the  menu  options  can 
help  the  users  understand  where  the  options  can  take  them.  While  most  of  the  menu  options  used  very 
clear  terminology,  we  suggest  changing  names  of  two  menu  options:  Log  to  Log  Entries,  and  Close  to 
Choose  Log.  This  tells  the  user  more  explicitly  what  they  will  see  if  they  choose  these  options.  Many 
users  expressed  a  desire  to  open  multiple  logs  without  closing  the  log  they  are  looking  at.  To  meet  this 
requirement  we  recommend  the  Choose  Log  menu  option  popping  up  a  window  where  the  users  can 
choose  a  log.  This  easily  allows  the  user  to  open  multiple  log  windows. 
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Using  the  titles  for  the  content  pages  that  are  consistent  with  the  highlighted  menu  option  also  helps  the 
users  to  check  that  they  are  in  the  right  place.  This  was  done  on  all  the  pages  except  for  the  Log  Entries 
page.  We  suggest  adding  a  title  to  this  page  as  well. 

Tell  the  user  how  to  get  back 

The  same  design  suggestions  that  orient  users  will  also  help  them  go  back.  However,  if  there  is  a  sequence 
of  screens  that  the  user  has  to  navigate  through  to  perform  a  function  (like  creating  a  new  search  or 
headline),  the  Back  links  have  to  be  provided  on  each  page.  In  general  such  long  chains  of  screen  should 
be  avoided  since  they  are  hard  to  navigate,  and  hide  the  functionality  from  the  users. 


Expose  the  functionality  to  the  user 

There  were  many  comments  from  users  about  not  being  able  to  find  different  functions  of  CommandNet, 
like  editing  entries,  creating  a  category,  etc.  These  users  knew  they  had  access  to  this  functionality,  but  it 
was  not  exposed  to  them  in  the  interface  and  they  had  to  navigate  through  many  pages  before  they  could 
find  what  they  needed.  To  address  this  problem,  it  is  necessary  to  expose  as  much  of  functionality  as 
possible  to  the  user  in  the  menu  and  in  the  content  of  the  page. 

A  lot  of  functionality  is  concealed  in  the  Admin  menu  option.  We  suggest  revealing  that  functionality 
directly  in  the  main  menu,  or  in  the  content  of  the  page  as  appropriate.  For  example.  Change  Password, 
New  Log,  Manage  Users,  Add  Category,  etc.  can  all  be  menu  options  for  those  users  who  have 
permissions  to  use  these  functions.  However,  the  editing  entries  functionality  would  be  best  exposed  as 
“edit”  buttons  to  the  left  of  each  entry  because  the  users  usually  try  to  edit  entries  while  reading  them. 

Please  note  that  only  the  users  who  have  the  editing  permissions  should  see  these  “edit”  buttons.  When  the 
edit  button  is  clicked,  a  new  window  should  pop  up  with  the  form  where  the  users  can  make  their  changes. 
A  window  pop  up  will  allow  for  more  space  for  the  edit  form,  so  that  a  multi-line  text  field  can  be  used  to 
display  the  entries.  In  the  current  editing  interface,  the  entry  text  is  displayed  in  a  one-line  text  field,  and 
users  commented  that  the  large  entries  did  not  fit  and  could  not  be  edited  properly.  Also,  the  entries  are 
displayed  in  ‘oldest  first— newest  last’  order,  which  is  inconsistent  with  the  order  of  entries  on  the  Log 
Entries  Screen,  and  made  it  harder  for  users  to  find  the  entry  they  wanted  to  edit. 

Provide  a  way  to  get  to  other  pages 

One  of  a  menu’s  functions  is  to  provide  a  way  to  get  to  other  pages  and  show  the  users  which  pages  they 
can  reach.  As  discussed  in  the  previous  section,  if  the  menu  options  are  not  explicit,  the  users  are  not  aware 
they  can  do  certain  functions.  The  inverse  of  this  principle  is  also  true.  If  the  menu  options  are  visible, 
users  assume  that  they  can  perform  all  those  functions  and  will  be  confused  if  they  cannot  click  on  them. 

In  the  screen  where  the  user  chooses  a  log.  Log,  Headlines,  Search,  Preferences,  Close,  and  Refresh 
Enable/Disable  are  not  available  to  the  user,  but  they  show  up  in  the  menu.  When  users  cannot  click  on 
them,  they  may  not  only  be  confused,  but  may  also  assume  that  these  menu  options  are  not  functional  in 
general,  and  thus  will  not  try  to  click  on  them  in  other  pages.  It  is  important  to  present  the  users  with  all 
and  only  the  options  available  to  them  at  each  point  during  navigation. 

Preferences 

The  preferences  interface  was  very  straightforward  and  easy  to  use.  The  problem  was  not  in  setting 
preferences,  but  in  keeping  them.  We  had  comments  from  users  that  if  they  used  the  refresh  button  on  their 
browser,  their  preferences  would  be  reset  back  to  default.  We  recommend  that  the  user’s  preferences 
should  never  be  changed  to  default  without  the  user  explicitly  resetting  them.  Even  if  the  user  logs  out  and 
then  logs  back  in,  preferences  should  stay  the  way  they  were  set. 

Look  and  Feel 

Overall,  the  look  and  feel  of  CommandNet  was  very  good,  and  users  commented  on  what  a  nice  application 
it  was.  We  have  some  minor  suggestions  that  can  improve  the  look  and  feel  of  CommandNet.  On  the  Log 
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Entries  page,  the  headings  in  the  table  that  contains  the  entries  should  align  with  the  content,  and  there 
should  be  space  between  the  headings.  In  the  current  interface  it  looks  like  there  are  two  columns,  “#  Imp 
Time  Entry”  and  “Cat  Name  Date”  (Eigure  4).  This  made  it  hard  for  users  to  figure  out  which  data  is 
displayed  in  which  column. _ 


Time 

Name  DatO 

-Il4b.rr^ry 
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3  P15:47'  33Z 

immediate  entrv 

laurie  l8Sep2ClOO 
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another  Laurie  eearctr 

iaurie  1 7Sep2OO0 

6  16:19-292 

Helen  is  making  a  link  $ 

laurie  l7Sep2009 

5  20:44  19Z 

Here'S  a  good  website  on  Strong 
Angel  9 

laurie  3lAug2000 

4  020:41:532 

LCDR  Teague  assumes  the  watch. 
Lt  Brooks  is  ABWC 

shift  laune  3lAug2000 
Change 

3  18:07  372 

Look  here  for  new  and  exciting 

laurie  17Aug2000 

Figure  4:  Current  CommandNet  view  of  Log  Entries  page.  Suggestion:  Align  headings  of  the  table 
with  the  content,  and  add  some  space  between  columns 

The  entry  forms  should  have  all  the  fields  aligned  on  the  left  and  their  text  labels  aligned  on  the  right  as 
shown  in  Eigure  5.  This  applies  not  only  to  the  creating  new  entry  form,  but  to  all  the  forms  in 
CommandNet. 


Importance: 

Routine 

Category: 

none  ^ 

URL  Link: 

URL  Desc.: 

Figure  5:  Suggested  form  alignment 

Many  HTML  forms  in  CommandNet  (Login  form,  new  search  form,  etc.)  could  benefit  from  the  cursor 
being  in  the  first  field  where  the  user  should  be  typing.  All  the  form  elements  should  be  labeled.  We 
suggest  putting  the  “New  Entry:”  label  above  the  multi-line  text  field  where  the  user  makes  a  new  entry. 
All  of  the  other  form  fields  were  labeled  very  clearly. 


Search  and  Headlines 

Search  and  headlines  are  what  we  consider  the  advanced  features  of  ConnnandNet.  Eew  users  took 
advantage  of  them  in  our  study,  but  that  does  not  mean  that  these  features  will  not  be  of  interest  to  the  users 
when  CommandNet  is  used  over  a  longer  period  of  time.  We  have  the  following  design  suggestions  on 
how  to  improve  the  interface  for  search  and  headlines  to  help  users  get  more  value  out  of  them. 

Put  new  search  and  saved  searches  on  the  same  screen  to  avoid  unnecessary  navigation  (Eigure  6).  The 
same  applies  to  new  headline  and  saved  headlines.  Add  the  title  “Saved  Searches”  or  “Saved  Headlines” 
above  the  saved  searches  or  headlines  to  explain  them  to  the  user  (Eigure  6).  Search  the  headlines  as  well 
as  entries,  or  specify  that  only  the  entries  can  be  searched. 

When  saving  searches,  ask  only  for  the  title  (because  the  description  is  usually  just  a  repetition  of  the  title). 
Save  the  search  terms  and  display  them  in  the  table  where  saved  searches  are  displayed,  instead  of 
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“Description”  column,  so  that  the  users  know  exactly  the  basis  of  the  search.  It  would  also  be  very 
beneficial  to  allow  users  to  edit  and  delete  their  searches. _ 

New  Searchi 


[Search  Area  goes  here] 


Saved  Searches 


1 

Time 

13: 09:^9  Z 

TttJe 

Laurie's  entdes 

Description 

entries  vdEh  Laurie's  name 

Name 

laurie 

Date 

17AU92000 

^  2 

search  for  "helen'" 

entries  v^th  "defen"  in  ihem 

laurie 

17Sep2000 

Figure  6:  Proposed  layout  of  search  that  avoids  unnecessary  navigation 

In  the  current  headlines  interface,  adding  an  entry  to  a  headline  after  entering  the  title  and  description  will 
erase  the  title  and  description.  We  suggest  a  “wizard”  approach  to  this  interface.  Wizards  are  interfaces 
that  take  the  user  through  a  process  step  by  step  providing  a  “back”  button  and  a  “next”  button.  In  this 
case,  the  first  step  is  to  select  the  entries  to  add  to  the  headline.  The  second  step  is  to  enter  the  title  and 
description.  If  this  process  is  broken  down  into  its  steps,  it  will  help  the  user  create  new  headlines. 
Another  issue  with  the  current  headline  interface  is  not  being  able  to  remove  entries  while  creating  a 
headline  (before  the  headline  is  submitted).  If  an  entry  is  added  in  error,  the  only  way  to  correct  this 
mistake  is  to  start  all  over  and  create  a  new  headline.  There  is  also  a  need  for  editing  the  searches  and 
headlines  after  they  are  created.  Only  some  users  should  have  permissions  to  do  so. 

We  believe  that  combining  the  search  and  headline  interfaces  can  be  very  beneficial.  During  the  usability 
tests  users  commented  how  they  would  like  to  create  a  headline  out  of  the  search  results.  We  suggest  that 
after  getting  search  results,  users  should  have  an  option  to  create  a  new  headline  out  of  resulting  entries. 
When  the  user  creates  a  new  headline  by  going  to  the  Headline  option  on  the  menu,  she  should  be  able  to 
search  for  the  entries  to  add  to  the  headline. 

Links 

Adding  links  to  entries  was  a  very  well  designed  feature.  It  is  very  useful  and  easy  to  use,  and  we  have 
very  positive  feedback  about  it  from  the  users. 


Security  and  the  Admin  interface 

In  CommandNet  users  have  User  permissions  or  Admin  permissions.  Having  only  User  permissions  allows 
the  user  to  create  other  users  with  Admin  permissions.  This  is  a  security  vulnerability  because  it  allows  any 
user  to  get  complete  access  to  the  system.  In  addition  to  enabling  and  enforcing  a  more  secure  permissions 
policy,  we  also  recommend  creating  admin  levels  such  that  a  user  can  be  an  administrator  of  one  log,  but 
not  all.  In  the  current  version  of  CommandNet,  an  admin  user  has  full  rights  to  everything  within 
CommandNet. 

There  are  navigation  issues  with  the  current  Admin  interface  as  well.  When  the  user  enters  the  system  she 
can  choose  a  log  or  can  choose  to  do  admin  functions.  If  she  chooses  to  do  the  admin  functions,  there  is  no 
navigation  provided  to  get  back  to  Choose  Log  screen.  As  suggested  before,  the  Close  link  on  the  menu 
should  be  renamed  Choose  Log.  It  should  be  highlighted  when  the  user  is  in  the  Choose  Log  screen,  and 
the  user  should  be  able  to  click  on  it  to  get  back  to  Choose  Log  screen.  This  is  consistent  with  the  point 
made  in  the  Navigation  section  of  this  report,  that  functionality  should  not  be  buried  in  the  menu.  Instead 
of  having  a  single  admin  link,  there  should  be  distinct  links  to  manage  users,  change  password,  etc. 
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Editing  the  Access  Control  List  is  available  in  the  Choose  Log  screen  only  before  any  log  is  chosen.  We 
observed  several  users  not  being  able  to  find  how  to  add  others  to  the  ACL  because  they  had  already 
chosen  a  log.  This  functionality  should  be  available  once  the  log  is  chosen  as  well  as  in  the  Choose  Log 
screen. 

When  users  clicks  on  the  Admin  menu  option,  they  are  presented  with  a  variety  of  possible  functions,  each 
with  an  icon  and  descriptive  text.  Only  the  icon  is  a  link,  the  text  is  not.  This  suggests  to  users  that  the 
option  might  not  be  available  to  them  because  they  cannot  click  on  it.  We  recommend  making  both  the 
icon  and  the  text  into  a  link. 


Chat 

There  were  many  requests  from  users  for  Chat  functionality.  CommandNet  lends  itself  to  chat,  and  it  came 
to  the  rescue  during  the  Strong  Angel  exercise  when  other  lines  of  communication  were  not  available. 
During  RIMPAC,  the  Chief  of  Staff  made  a  policy  decision  to  use  CommandNet  only  for  significant  events 
and  lessons  learned.  Many  users  commented  that  although  they  should  not  use  the  logs  for  chat,  a  chat 
feature  would  be  very  useful.  To  meet  the  need  for  chat  we  recommend  installing  a  chat  application  for 
people  to  use  in  conjunction  with  CommandNet. 


Name 

The  ship’s  radio  network  was  also  called  CommandNet.  This  made  the  reference  to  CommandNet 
ambiguous,  since  one  could  be  referring  either  to  the  radio  network  or  collaborative  log. 


References 

Nygren,  E.,  &  Allard,  A.  (1996).  Between  the  Clicks”  Skilled  Users  Scanning  of 
Pages.  Designing  for  the  Web:  Empirical  Studies.  Human  Eactors  and  the 
Web/HTML  Conference.  Sandia  National  Laboratories,  Albuquerque,  NM. 


Cl 


The  MITRE  Corporation 


